If you are like self hoster like me and you self host kubernetes at home, there is a high chance that you also self host pi-hole. Pi-hole in short is a DNS server, with a ability to block DNS request that may be malicious (malware, ads).
It's simply for blocking ads, but it has much more features - DHCP, conditional forwarding, DNSEC and more.
But enough about pi-hole, which I recommend to try out! What happened to me was that my little pi-hole DNS server running on rock64 was getting loads of traffic thanks to the k3s nodes, and docker machines runing in my network. Some spikes went to 16k request per 10 minutes. Yikes. That's a lot traffic for small network at my home.
Anyway by design pi-hole allows specific amount of traffic and then if there just too many requests in short amount of time, some traffic gets refused which means they get response
REFUSED. I like the simplicity of the answer btw. To check if this is happening to you there are simply two indicators:
- you find in query logs request with status
toolsin "Pi-hole diagnosis", you'll find many messages that tell you:
Maximum number of concurrent DNS queries reached (max: 150)
How to fix this
We know that pi-hole blocks the requests, which can break soo many thinks. Just think about it. If some critical DNS is refused it can break coredns for kubernetes, spam you with alert message that some service is down, break you docker containers and so much more.
Now to fix this after some hours of googling-fu I found this solution. First you need to some configs. In my case I run pi-hole in container I had to exec inside container
docker exec -ti pihole bash and install editor of my choice.
After that you need to set
Then go to the directory
/etc/dnsmasq.d where I created the file
99-maxsettings.conf. And inside this config file I have two following settings:
First options disables the 150 limit for DNS queries for dnsmasq. And the second option is optional but in my case I noticed that I have one specify record requested over 166051 times. This options simply excludes the record from DNS rebinding protection and helps with the spam. I also had to restart coredns deployment in k3s, after this.
After I set these options I was getting much more requests thanks to disabling the limit at first. Almost 35k request per 10 minutes! But after couple hours my pi-hole went into much calmer state. Much less requests, no alerts, and my network was working fine until the next problem.
Hope this helps you and take care fellow self hoster!