bug: Changing FluentBit labels causes immutable selector error

[ruben-chainalysis]

Describe the issue

When labels are added or modified in the FluentBit spec.labels field after the DaemonSet has been created, the operator attempts to update the DaemonSet's immutable spec.selector field, causing continuous reconciliation failures.

ERROR Reconciler error {"controller": "FluentBit", "error": "DaemonSet.apps \"fluent-bit\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app\":\"fluentbit\", \"fluentbit.fluent.io/enabled\":\"true\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable"}

To Reproduce

1. Deploy fluent-operator with a FluentBit CR in DaemonSet mode with initial labels:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBit
metadata:
  name: fluent-bit
spec:
  labels:
    app: fluentbit

2. Wait for DaemonSet to be created and running
3. Add a new label to the FluentBit CR:

spec:
  labels:
    app: fluentbit
    fluentbit.fluent.io/enabled: "true"  # New label added

4. Apply the updated CR: kubectl apply -f fluentbit.yaml

5. Observe continuous reconciliation failures in operator logs:

ERROR Reconciler error {"controller": "FluentBit", "error": "DaemonSet.apps \"fluent-bit\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app\":\"fluentbit\", \"fluentbit.fluent.io/enabled\":\"true\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable"}

Expected behavior

When labels are changed in the FluentBit CR, the operator handles the change without causing continuous reconciliation failures. The user's intended label changes should be applied to the DaemonSet (or the operator should provide clear guidance on how to proceed).

Your Environment

- Fluent Operator version: v3.4.0
- Container Runtime: containerd
- Operating system: Amazon Linux 2023
- Kernel version: 6.12.x

How did you install fluent operator?

Installed via Helm chart with FluxCD

Additional context

Current workaround: Manually delete the DaemonSet to force recreation:

kubectl delete daemonset fluent-bit -n <namespace>

The operator immediately recreates it with the updated selector.

Related: Issue #1424 discusses inconsistent label/annotation behavior in FluentBit CR. Both issues involve how labels in the FluentBit CR affect the DaemonSet.

Impact:

• Reconciliation fails continuously
• Adding/removing labels in FluentBit CR requires manual intervention
• Other legitimate DaemonSet updates are also blocked when selector mismatch exists

[ruben-chainalysis]

I've built an image that instead of freezing when these changes happen, just deletes and creates a new daemonset. Not sure if that's the best solution, can you see any problems with that approach? I will happily submit a PR following your advice.

[patrick-stephens]

The main issue would be with losing any data in flight whilst recreating - which in turn relies on the configuration slightly (e.g. if there is no persistence then you may lose anything in memory).

[ruben-chainalysis]

Thanks for your feedback, I'm testing some changes to deal with annotations in a similar way so they are an alternative. One thing I haven't solved by adding the annotations capability is what happens when the labels change (as reported initially in this issue). By adding the annotations we have an option to avoid the error, but what if we still hit it? Shouldn't the operator do something better than logging an error message and staying in a silent error (log) loop that doesn't end in a restart?