Failed to create connection: expected a single Running pod with IP address

[Smrutiranjanmohanty]

Hi guys. I don't know what is the wrong I am doing. Because I have successfully migrated nginx ingress to nginx gateway in dev and stage gke cluster. But when I am instlling the gateway controller and creating gateway in prod, I am getting isuue
Here is the error log of gateway pod

time=2025-12-09T16:45:24.743Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unauthenticated desc = expected a single Running pod with IP address "20.1.6.1", but found 0" correlation_id=6c695e3d-d51d-11f0-b52a-8a6e1f9b3250 server_type=command
time=2025-12-09T16:45:36.271Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unauthenticated desc = expected a single Running pod with IP address "20.1.6.1", but found 0" correlation_id=6c695e3d-d51d-11f0-b52a-8a6e1f9b3250 server_type=command
time=2025-12-09T16:45:54.843Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unauthenticated desc = expected a single Running pod with IP address "20.1.6.1", but found 0" correlation_id=6c695e3d-d51d-11f0-b52a-8a6e1f9b3250 server_type=command
time=2025-12-09T16:46:07.550Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unauthenticated desc = expected a single Running pod with IP address "20.1.6.1", but found 0" correlation_id=6c695e3d-d51d-11f0-b52a-8a6e1f9b3250 server_type=command
time=2025-12-09T16:46:31.152Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unauthenticated desc = expected a single Running pod with IP address "20.1.6.1", but found 0" correlation_id=6c695e3d-d51d-11f0-b52a-8a6e1f9b3250 server_type=command
time=2025-12-09T16:46:42.435Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unauthenticated desc = expected a single Running pod with IP address "20.1.6.1", but found 0" correlation_id=6c695e3d-d51d-11f0-b52a-8a6e1f9b3250 server_type=command

[nginx-bot]

Hi @Smrutiranjanmohanty! Welcome to the project! 🎉

Thanks for opening this issue!
Be sure to check out our Contributing Guidelines and the Issue Lifecycle while you wait for someone on the team to take a look at this.

[salonichf5]

Hi @Smrutiranjanmohanty, thanks for your interest in NGINX Gateway Fabric 🙌

From the log:

rpc error: code = Unauthenticated desc = expected a single Running pod with IP address "20.1.6.1", but found 0

the command server is rejecting the connection because it can’t find any Running pod with IP 20.1.6.1 in the cluster.

In your prod setup, do you have 20.1.6.1 configured anywhere and does kubectl get pods -A -o wide | grep 20.1.6.1 show a pod with that IP?

[Smrutiranjanmohanty]

Their is no pod with specific ip

kubectl get pods -A -o wide | grep 20.1.6.1
nginx-gateway nginx-gateway-nginx-75c979499f-f4xvk 0/1 Running 0 107m 20.1.6.11 gke-gke-broadway-eco-broadway-ecom-pr-d593628c-hpgh

[jaswanthagfa]

I am sorry for Jumping into this conversation. I also have a same issue while installing the latest Helm chart. In My case it was my Nodeip where the Nginx Control plane pod is Hosted.

Logs from the dataplane pod

time=2025-12-09T20:17:19.265Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unauthenticated desc = expected a single Running pod with IP address \"10.xy.xx.77\", but found 0" correlation_id=ebabd09c-d538-11f0-97a7-969001e0b744 server_type=command

Logs from the ngf-nginx-gateway-fabric-69b784665c Pod

{"level":"error","ts":"2025-12-09T20:19:09Z","logger":"agentGRPCServer","msg":"error validating connection","error":"rpc error: code = Unauthenticated desc = expected a single Running pod with IP address \"10.xx.xx.77\", but found 0","stacktrace":"github.com/nginx/nginx-gateway-fabric/v2/internal/controller/nginx/agent/grpc/interceptor.(*ContextSetter).Unary.ContextSetter.Unary.func1\n\t/opt/actions-runner/_work/nginx-gateway-fabric/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor/interceptor.go:80\ngithub.com/nginx/agent/v3/api/grpc/mpi/v1._CommandService_CreateConnection_Handler\n\t/opt/actions-runner/_work/nginx-gateway-fabric/nginx-gateway-fabric/.gocache/github.com/nginx/agent/v3@v3.5.0/api/grpc/mpi/v1/command_grpc.pb.go:193\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/opt/actions-runner/_work/nginx-gateway-fabric/nginx-gateway-fabric/.gocache/google.golang.org/grpc@v1.76.0/server.go:1431\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/opt/actions-runner/_work/nginx-gateway-fabric/nginx-gateway-fabric/.gocache/google.golang.org/grpc@v1.76.0/server.go:1842\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/opt/actions-runner/_work/nginx-gateway-fabric/nginx-gateway-fabric/.gocache/google.golang.org/grpc@v1.76.0/server.go:1061"}

10.xy.xx.77 This is my Worker NodeIp where ngf-nginx-gateway-fabric-69b784665c is hosted.

Kubernetes Version: v1.30.4+rke2r1

We Use Calico and underlying Kubernetes Cluster is Rke2 Rancher based.

Rancher Version: v2.9.1

[sjberman]

Interesting...when the nginx agent connects to the control plane, we intercept the grpc context and extract some details, including the IP address. I had no idea that this could be a node IP and not the IP address of the Pod.

This is important because we use that IP address to determine which Pod that agent connection is actually coming from. When we can't find the pod associated with that IP address, you see the error above.

Now the question is how do we avoid that.

[sjberman]

#4351 could be related, with hostNetwork causing the pod IP to not be present. Although this bug gets further along in the process.

[jaswanthagfa]

nginx: service: type: NodePort nodePorts: - port: 32000 listenerPort: 80 - port: 32500 listenerPort: 8080 - port: 32443 listenerPort: 443 debug: true

Above is my custom values and I am not doing any patches on top of it.

This is my data plane deployment manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
    field.cattle.io/publicEndpoints: '[{"addresses":["10.xx.xx.150"],"port":32000,"protocol":"TCP","serviceName":"nginx-gateway:colors-gateway-nginx","allNodes":true}]'
  creationTimestamp: "2025-12-09T19:41:45Z"
  generation: 21708
  labels:
    app.kubernetes.io/instance: ngf
    app.kubernetes.io/managed-by: ngf-nginx
    app.kubernetes.io/name: colors-gateway-nginx
    gateway.networking.k8s.io/gateway-name: colors-gateway
  name: colors-gateway-nginx
  namespace: nginx-gateway
  resourceVersion: "120290213"
  uid: 3b5a31a0-74a0-43fe-a336-001a2c408cd5
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: ngf
      app.kubernetes.io/managed-by: ngf-nginx
      app.kubernetes.io/name: colors-gateway-nginx
      gateway.networking.k8s.io/gateway-name: colors-gateway
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/port: "9113"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: ngf
        app.kubernetes.io/managed-by: ngf-nginx
        app.kubernetes.io/name: colors-gateway-nginx
        gateway.networking.k8s.io/gateway-name: colors-gateway
    spec:
      automountServiceAccountToken: true
      containers:
      - args:
        - debug
        command:
        - /agent/entrypoint.sh
        image: ghcr.io/nginx/nginx-gateway-fabric/nginx:2.2.1
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 80
          name: port-80
          protocol: TCP
        - containerPort: 9113
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 3
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsUser: 101
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/nginx-agent
          name: nginx-agent
        - mountPath: /var/run/secrets/ngf
          name: nginx-agent-tls
        - mountPath: /var/run/secrets/ngf/serviceaccount
          name: token
        - mountPath: /var/log/nginx-agent
          name: nginx-agent-log
        - mountPath: /var/lib/nginx-agent
          name: nginx-agent-lib
        - mountPath: /etc/nginx/conf.d
          name: nginx-conf
        - mountPath: /etc/nginx/stream-conf.d
          name: nginx-stream-conf
        - mountPath: /etc/nginx/main-includes
          name: nginx-main-includes
        - mountPath: /etc/nginx/events-includes
          name: nginx-events-includes
        - mountPath: /etc/nginx/secrets
          name: nginx-secrets
        - mountPath: /var/run/nginx
          name: nginx-run
        - mountPath: /var/cache/nginx
          name: nginx-cache
        - mountPath: /etc/nginx/includes
          name: nginx-includes
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - /usr/bin/gateway
        - initialize
        - --source
        - /agent/nginx-agent.conf
        - --destination
        - /etc/nginx-agent
        - --source
        - /includes/main.conf
        - --destination
        - /etc/nginx/main-includes
        - --source
        - /includes/events.conf
        - --destination
        - /etc/nginx/events-includes
        env:
        - name: POD_UID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        image: ghcr.io/nginx/nginx-gateway-fabric:2.2.1
        imagePullPolicy: IfNotPresent
        name: init
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsUser: 101
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /agent
          name: nginx-agent-config
        - mountPath: /etc/nginx-agent
          name: nginx-agent
        - mountPath: /includes
          name: nginx-includes-bootstrap
        - mountPath: /etc/nginx/main-includes
          name: nginx-main-includes
        - mountPath: /etc/nginx/events-includes
          name: nginx-events-includes
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
        runAsNonRoot: true
        sysctls:
        - name: net.ipv4.ip_unprivileged_port_start
          value: "0"
      serviceAccount: colors-gateway-nginx
      serviceAccountName: colors-gateway-nginx
      terminationGracePeriodSeconds: 30
      volumes:
      - name: token
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              audience: ngf-nginx-gateway-fabric.nginx-gateway.svc
              expirationSeconds: 3600
              path: token
      - emptyDir: {}
        name: nginx-agent
      - configMap:
          defaultMode: 420
          name: colors-gateway-nginx-agent-config
        name: nginx-agent-config
      - name: nginx-agent-tls
        secret:
          defaultMode: 420
          secretName: colors-gateway-nginx-agent-tls
      - emptyDir: {}
        name: nginx-agent-log
      - emptyDir: {}
        name: nginx-agent-lib
      - emptyDir: {}
        name: nginx-conf
      - emptyDir: {}
        name: nginx-stream-conf
      - emptyDir: {}
        name: nginx-main-includes
      - emptyDir: {}
        name: nginx-events-includes
      - emptyDir: {}
        name: nginx-secrets
      - emptyDir: {}
        name: nginx-run
      - emptyDir: {}
        name: nginx-cache
      - emptyDir: {}
        name: nginx-includes
      - configMap:
          defaultMode: 420
          name: colors-gateway-nginx-includes-bootstrap
        name: nginx-includes-bootstrap
status:
  conditions:
  - lastTransitionTime: "2025-12-09T19:41:45Z"
    lastUpdateTime: "2025-12-09T19:41:45Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2025-12-09T20:04:50Z"
    lastUpdateTime: "2025-12-09T20:04:50Z"
    message: ReplicaSet "colors-gateway-nginx-648bfb8bf" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 21708
  replicas: 2
  unavailableReplicas: 2
  updatedReplicas: 1

[sjberman]

I was able to reproduce the issue by turning on hostNetwork, specifying the following in my NginxProxy resource:

kubernetes:
    deployment:
        patches:
        - type: StrategicMerge
          value:
            spec:
              template:
                spec:
                  dnsPolicy: ClusterFirstWithHostNet
                  hostNetwork: true
                  securityContext:
                    sysctls: []

The question we have to figure out is how to support this. Maybe have to rethink how we use the IP address internally, I think the main uses are:

• internal tracking of all agent connections (could probably use hostname as the key for this instead of IP address)
• getting the pod to verify that it's the only pod with that IP address, so that nobody is spoofing the service account token presented to the control plane (maybe this is overkill or there's a better way to do it)