NS queries by HAOS

[Zixim]

Does anyone have an idea why my HAOS instance would be doing tens of thousands of these queries per day :

dnsmasq[171845]: query[NS] . from 192.168.11.5

This is a snippet from the log on pihole.
Strangely, the logs of HAOS show none of these queries... are they trying to hide something ?

[bentasker]

That's most likely generated by CoreDNS's health checks - https://github.com/coredns/coredns/blob/b7b62b38efe281ecd17cad63765725d3ed4cbf18/plugin/forward/health.go#L78

If you're seeing that many, either coreDNS thinks something's failing, or there's some kind of feedback loop (either is possible tbh).

Are you able to grab a copy of the currently active config? I might be able to tell more from that.

[Zixim]

# docker exec -it hassio_dns /bin/bash
bash-5.1# cat /usr/share/tempio/corefile
.:53 {
    log {{ if not .debug }}{
        class error
    }{{ end }}
    errors
    loop
    {{ if .debug }}debug{{ end }}
    hosts /config/hosts {
        fallthrough
    }
    template ANY AAAA local.hass.io hassio {
        rcode NOERROR
    }
    mdns
    forward . {{ join " " .servers }} {{ if len .locals | eq 0 }}dns://127.0.0.11{{ else }}{{ join " " .locals }}{{ end }} dns://127.0.0.1:5553 {
        except local.hass.io
        policy sequential
        health_check 1m
    }
    fallback REFUSED,SERVFAIL,NXDOMAIN . dns://127.0.0.1:5553
    cache 600
}

.:5553 {
    log {{ if not .debug }}{
        class error
    }{{ end }}
    errors
    {{ if .debug }}debug{{ end }}
    forward . tls://1.1.1.1 tls://1.0.0.1 {
        tls_servername cloudflare-dns.com
        except local.hass.io
        health_check 5m
    }
    cache 600
}
bash-5.1#

[Zixim]

oops, that was the template of course...
now for the corefile :

bash-5.1# cat /etc/corefile
.:53 {
    log {
        class error
    }
    errors
    loop

    hosts /config/hosts {
        fallthrough
    }
    template ANY AAAA local.hass.io hassio {
        rcode NOERROR
    }
    mdns
    forward . dns://192.168.11.2 dns://192.168.11.2  {
        except local.hass.io
        policy sequential

    }

    cache 600
}

.:5553 {
    log {
        class error
    }
    errors

    forward . tls://1.1.1.1 tls://1.0.0.1 {
        tls_servername cloudflare-dns.com
        except local.hass.io

    }
    cache 600
}
bash-5.1#

[bentasker]

Hmmm, no healthchecks configured then.

However, from a quick scan over, it looks like CoreDNS will kick off a healthcheck if it perceives a query has failed - https://github.com/coredns/coredns/blob/b7b62b38efe281ecd17cad63765725d3ed4cbf18/plugin/forward/forward.go#L162

There are various reasons it might decide the query failed - https://github.com/coredns/coredns/blob/b7b62b38efe281ecd17cad63765725d3ed4cbf18/plugin/forward/connect.go#L88 - but they're all Layer3 (i.e. failure to connect, connection timeout, incomplete response etc etc)

What that means in practice, though, is

• HA tries to query foo.example.com
• Your local recursor receives the query and tries to resolve
• The authoritatives are having issues, so take a long time to respond (or never do)
• Eventually HA times out
• CoreDNS ultimately will kick of a round of healthchecks

I don't know how we get from that to 10's of thousands a day though - looking in my logs, I don't see the same behaviour - my config is basically the same as you've got.

I wonder if you having the same upstream appear twice is screwing with something? I assume your DHCP is setting them as first and second? Might be worth switching the addon to template rather than patch mode and adding that server once to see if it changes the behaviour.

[Zixim]

I set up this template now :

.:53 {
    log {
        class error
    }
    errors
    loop
    
    hosts /config/hosts {
        fallthrough
    }
    template ANY AAAA local.hass.io hassio {
        rcode NOERROR
    }
    mdns
    forward . dns://192.168.11.2  {
        except local.hass.io
        policy sequential
        
    }
    
    cache 600
}
.:5553 {
    log {
        class error
    }
    errors
    
    forward . tls://1.1.1.1 tls://1.0.0.1 {
        tls_servername cloudflare-dns.com
        except local.hass.io
        
    }
    cache 600
}

Line 15 being changed to forward . dns://192.168.11.2 .
I have no idea how the 2x reference to dns://192.168.11.2 got in there ... The "." lookups are now indeed gone.

[bentasker]

So now.... we wait :)

[Zixim]

aaaand...
Dec 4 21:16:48 dnsmasq[171845]: query[NS] . from 192.168.11.5
fwiw :

➜  ~ ha dns info   
host: 172.30.32.3
locals:
- dns://192.168.11.2
servers:
- dns://192.168.11.2
update_available: false
version: 2021.06.0
version_latest: 2021.06.0

corefile is still as per my previous...

doing a ha dns restart temporarily helps.

[bentasker]

Sounds like some queries are failing - have you had any other issues (or perhaps blocked some domains). Probably worth looking into in case it's exposing an issue.

In the codebase there's a conditional

			if f.maxfails != 0 {
				proxy.Healthcheck()
			}

So if that config var is 0 it'll not bother with the healthchecks.

That appears to map back to max_fails, so try adding max_fails 0 into the forward sections:

    forward . dns://192.168.11.2  {
        except local.hass.io
        policy sequential
        max_fails 0 
    }
...
    forward . tls://1.1.1.1 tls://1.0.0.1 {
        tls_servername cloudflare-dns.com
        except local.hass.io
        max_fails 0
    }

Nothing should be using the bottom one, but I'd be inclined to add it there anyway as a belt-and-braces thing.

[Zixim]

have you had any other issues (or perhaps blocked some domains)
The HAOS host hasn't had any requests blocked, pretty easy to verify in PiHole.

Meanwhile, things are behaving normally - for nearly 24H, I'll try your edit if/when the thing kicks off another round of insane "." lookups.

[Zixim]

if/when the thing kicks off another round of insane "." lookups.

That would be now.