Cannot add custom network policies to Calico components

[sebhoss]

We just updated to the latest version of the operator (1.42) and the latest version of calico (3.32 OSS). 2 things broke for us:

1. Prometheus can no longer scrape metrics from the calico-kube-controllers pods (but felix/typha seem to work fine)
2. Whisker can no longer be reached through our Ingress

Since both endpoints are still working fine when using kubectl port-forward, we noticed #4438 which talks about adjusting network policies, but does not go into detail how we are supposed to adjust network policies.

What we had before was something like this:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: whisker-ingress
  namespace: calico-system
spec:
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
          podSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/instance: ingress-nginx
              app.kubernetes.io/component: controller
      ports:
        - port: 8081
          protocol: TCP
  podSelector:
    matchLabels:
      app.kubernetes.io/name: whisker
  policyTypes:
    - Ingress

I've tried converting that to a CNP like this:

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: whisker-ingress
  namespace: calico-system
spec:
  ingress:
    - action: Allow
      protocol: TCP
      source:
        namespaceSelector: kubernetes.io/metadata.name == 'ingress-nginx'
        selector: app.kubernetes.io/name == 'ingress-nginx' && app.kubernetes.io/instance == 'ingress-nginx' && app.kubernetes.io/component == 'controller'
      destination:
        ports:
          - 8081
  selector: k8s-app == 'whisker'
  tier: calico-system
  types:
    - Ingress

However, it is not possible to apply this network policy because the API server rejects it with Error from server (NotFound): networkpolicies.projectcalico.org "whisker-ingress" not found. Likewise changing the name of the policy to calico-system.whisker-ingress or removing the spec.tier field entirely results in the same error message. We can apply CNPs in other namespaces without problem, but it seems something is rejecting our creation of new network policies in the calico-system namespace.

[sebhoss]

sooo... we now did the steps in https://docs.tigera.io/calico/latest/operations/crd-migration and can add network policies to the calico-system namespace again. I cannot check, but I'm now suspecting that the network policies created by the operator prohibit calico-apiserver from being reached? Since we now do not have to use calico-apiserver anymore, everything is working fine again.

[sebhoss]

Just thinking out loud here, but could the operator generate all CNPs with spec.order? Right now we have:

$ kubectl --namespace calico-system get cnp
Warning: short name "cnp" could also match lower priority resource clusternetworkpolicies.policy.networking.k8s.io
NAME                                   TIER            ORDER
calico-system.apiserver-access         calico-system   1
calico-system.calico-webhooks          calico-system   1
calico-system.default-deny             calico-system   
calico-system.goldmane                 calico-system   
calico-system.kube-controller-access   calico-system   1
calico-system.whisker                  calico-system   

Neither goldmane/whisker nor the default-deny policy have an order defined. I'm not entirely sure what's the best way to introduce custom network policies here. If default-deny would have order: 100 and all other CNPs have order: 10, I could add my own policies with order: 1 to ensure that my policies are evaluated first.