Reconciliation Robustness and Startup Behavior

[lukidoescode]

Motivation

The operator must be resilient to restarts, network partitions, lldap downtime, and concurrent CRD changes. It must not thrash external state on restart and must handle all ordering dependencies gracefully.

Definition of Done

• Startup sequence:
  1. Verify lldap connectivity (fail-fast behavior per Ticket Operator Core / Controller Setup #8).
  2. Create/update virtual LldapGroup CRDs for built-in groups (Ticket Group Reconciler #13 provides this logic).
  3. Start controllers.
  4. On first reconcile of each resource: check observedGeneration. If it matches metadata.generation, skip spec-field processing (no change since last successful run). Password handling still runs its own hash-based comparison (Ticket User Reconciler #11). This prevents mass-updating lldap on restart.
• Idempotency guarantees:
  • Every mutation is guarded by a diff check: only call lldap API when the desired state differs from the last-known state.
  • The "last-known state" is encoded in the resource's status (uuid, groupId, passwordHash, observedGeneration).
• Ordering/dependency handling:
  • LldapMembership waits for both referenced user and group to be Ready.
  • LldapUser/LldapGroup with custom attributes waits for the LldapAttributeSchema to be Ready.
  • Dependencies are expressed via requeue + condition, not hard blocking.
• Failure resilience:
  • lldap unavailable: all reconcilers requeue with exponential backoff (5s, 10s, 30s, 60s, 5min cap).
  • JWT token expired mid-reconcile: client auto-refreshes and retries.
  • Kubernetes API unavailable: kube-rs controller runtime handles this internally (watch reconnection).
  • Partial failure: if creating a user succeeds but setting password fails, status reflects the partial state. Next reconcile continues from where it left off.
• Concurrent safety:
  • Multiple CRD changes to the same resource in quick succession: the controller deduplicates and reconciles the latest state.
  • Race between membership reconciler and user/group reconciler: membership retries until dependencies are ready.
• Resource efficiency:
  • After successful reconciliation, use Action::await_change() (no periodic requeue) for resources in steady state.
  • Use Server-Side Apply for status patches to minimize conflicts. This applies to ALL reconcilers (Tickets Attribute Schema Reconciler #10-14) — SSA should be the default status patching mechanism from the start, not a retroactive change.
  • Secret watches use label/field selectors where possible to reduce API server load.
• Documentation: Document the startup sequence, dependency ordering, and error recovery behavior in the Troubleshooting chapter. Document what each status condition means and how to resolve common issues.

Technical Details

• Exponential backoff: implement as a helper that maps error categories to requeue durations. Use the error_policy function.
• Status as state cache: the status subresource serves as the reconciler's memory of what it last wrote to lldap. This avoids querying lldap on every reconcile cycle.
• Generation tracking: metadata.generation only increments on spec changes (not status or metadata changes). This is the correct signal for "user changed something".
• Consider using a ResourceVersion-based optimistic lock for status updates to detect stale writes.