Intermittent DNS issues (ever 20 secs)

[ed-pai]

UPDATE: This was causes by GitLab ingresses (deployed via helm) to flap the public IPs, causing KPC to reconcile ever 20 secs and restart unbound service. This was an environment problem, not a problem with the controller.

It looks like this repo isn't maintained anymore, but creating this issue in case anyone struggles with the same issues I've been having for the past year. I use PFSense and run a few different K8S clusters for different services and testing. I've got this controller deployed in a couple of the clusters.

TL;DR:

This controller causes the PFSense DNS to restart multiple times a minute, which causes DNS to be unavailable for a few seconds. Recommend not using this controller and instead going with ExternalDNS or something like that.

Problem

I found out today that the intermittent DNS issues I've been having for the past year were caused by the way the controller does XMLRPC updates. In my cluster, they happen every ~18-22 secs (as seen in the system logs):

Dec  4 14:53:59 pfSense1 php-fpm[2348]: /xmlrpc.php: Configuration Change: (system)@172.16.10.36: Merged in config (unbound sections) from XMLRPC client.
Dec  4 14:54:17 pfSense1 php-fpm[71702]: /xmlrpc.php: Configuration Change: (system)@172.16.10.36: Merged in config (unbound sections) from XMLRPC client.

This causes the unbound service to restart approx 4 secs after the update:

Dec  4 14:54:03 pfSense1 unbound[11652]: [11652:0] notice: Restart of unbound 1.23.0.
Dec  4 14:54:21 pfSense1 unbound[71471]: [71471:0] notice: Restart of unbound 1.23.0.

There were hundreds of these logs. The DNS Services takes a few secs to fully restart and doesn't respond to requests during that time. The more of the controllers you deploy in your environment, the worse the problem gets.

Possible Alternatives

The Kubernetes SIG External-DNS has an Unbound Webhook that is capable of remotely updating PFSense. This requires enabling unbound-remote and opening the port externally on the PFSense, but it's doable.

[travisghansen]

I’m here! I just consider the project mostly complete but still use it regularly across many clusters myself.

Can you tell what k8s resource is updating that is causing so many frequent pfsense calls? Something must be getting modified on some resource(s) in your cluster triggering update events to happen.

Maybe send over your config and I can have a better cluster clue what may be at play. Also the logs may provide some info also..

[ed-pai]

Oh, sorry. Didn't realize you were still monitoring.
I think the problem is that I've got ~8 clusters deployed with it and stand-up services constantly.
I've installed it with the helm chart:

helm upgrade \
  --install \
  kpc \
  kubernetes-pfsense-controller/kubernetes-pfsense-controller \
  --version $TOOLS__KPC_VERSION \
  --create-namespace \
  --namespace $TOOLS__KPC_NS \
  --values values.yaml \
  --set pfsense.url="$TOOLS__PFSENSE_URL" \
  --set pfsense.username="$TOOLS__PFSENSE_USERNAME" \
  --set pfsense.password="$TOOLS__PFSENSE_PASSWORD" \
  --set pfsense.url="$TOOLS__PFSENSE_URL" \
  --set config.controller-id="$TOOLS__PFSENSE_CONTROLLER_ID" 

And mostly keep the values file the same:

# Default values for kubernetes-pfsense-controller.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

pfsense:
  url: 
  insecure: true
  username: ""
  password: ""

# you MUST copy the remainder of the config from example to meet your needs
# https://github.com/travisghansen/kubernetes-pfsense-controller/blob/master/deploy/config.yaml 
config: 
  controller-id: ""
  enabled: true
  plugins:
    metallb:
      enabled: true
      nodeLabelSelector:
      nodeFieldSelector:
      #configMap: "metallb-system/config"
      # pick 1 implementation
      #bgp-implementation: openbgp
      bgp-implementation: frr
      options:
#        openbgp:
          # pass through to config.xml
#          template:
#            md5sigkey:
#            md5sigpass:
#            groupname: metallb
#            row:
#              - parameters: announce all
#                parmvalue:
        frr:
          template:
            peergroup: metallb
    haproxy-declarative:
      enabled: true
    haproxy-ingress-proxy:
      enabled: true
      ingressLabelSelector:
      ingressFieldSelector:
      # works in conjunction with the ingress annotation 'haproxy-ingress-proxy.pfsense.org/enabled'
      # if defaultEnabled is empty or true, you can disable specific ingresses by setting the annotation to false
      # if defaultEnabled is false, you can enable specific ingresses by setting the annotation to true
      defaultEnabled: true
      defaultFrontend: http-80
      defaultBackend: traefik
      # by default anything is allowed
      #allowedHostRegex: "/.*/"
    pfsense-dns-services:
      enabled: true
      serviceLabelSelector:
      serviceFieldSelector:
      #allowedHostRegex: "/.*/"
      dnsBackends:
        dnsmasq:
          enabled: false
        unbound:
          enabled: true
    pfsense-dns-ingresses:
      enabled: true
      ingressLabelSelector:
      ingressFieldSelector:
      # works in conjunction with the ingress annotation 'dns.pfsense.org/enabled'
      # if defaultEnabled is empty or true, you can disable specific ingresses by setting the annotation to false
      # if defaultEnabled is false, you can enable specific ingresses by setting the annotation to true
      defaultEnabled: true
      #allowedHostRegex: "/.*/"
      dnsBackends:
        dnsmasq:
          enabled: false
        unbound:
          enabled: true
    pfsense-dns-haproxy-ingress-proxy:
      enabled: false
      #allowedHostRegex: "/.*/"
      dnsBackends:
        dnsmasq:
          enabled: false
        unbound:
          enabled: true
      frontends:
        http-80:
          hostname: http-80.k8s
        primary_frontend_name2:
          hostname: some-name.k8s

# do NOT set this higher than 1
# it is only an option to allow scaling to 0
replicaCount: 1

image:
  repository: docker.io/travisghansen/kubernetes-pfsense-controller
  pullPolicy: Always
  #pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "latest"

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

rbac:
  enabled: true

podAnnotations: {}

podSecurityContext: {}
  # fsGroup: 2000

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

[ed-pai]

I need to be able to monitor K8S Gateway resources now, so took me down the External-DNS path (since KPC doesn't currently support Gateway's I need to move away from it anyway.). While figuring out how to enable unbound-remote, I realized the DNS issue I've been having was a result of the XMLRPC hits on a regular basis.

[travisghansen]

Yeah you’re watching ingresses and services both it appears so something must tweaking those like crazy.

I have wondered what to do about gateway api, it is significantly more complex in structure compared to ingress so haven’t messed with it yet :(

[ed-pai]

I was considering writing my own controller in Go that uses the PFSense REST API, but I'm not sure if the service restarts when it's updated there as well.
Seems like ExternalDNS w/ unbound-remote webhook might be the best solution for both gateway and preventing unbound restarts.

[travisghansen]

Well, I think for the dns setup you’re trying to do external-dns is fine.

Alternatively I wouldn’t write a new controller per-se but rather a new out-of-tree provider for external-dns explicitly for pfsense.

[ed-pai]

Following up on this. I found out that we had 3 ingresses that keep changing IPs in my environment (GITLAB). This is causing KPC to reconcile and update the entries constantly, causing the intermittent outages on DNS.

I apologize for assuming it was an inherent problem with KPC. This looks like it was an environment specific problem on my end.

We ended up writing a webhook for ExternalDNS that uses the PFREST API & Token ( and supports Gateway), and will be releasing that OSS shortly.

Again, please accept my apology. Closing this ticket.

[travisghansen]

No problem! Feel free to share your solution here for others to see.