Observability (Metrics, Logging, Events)

[lukidoescode]

Motivation

Production operators require comprehensive observability. Cluster operators need to understand what the operator is doing, whether it's healthy, and diagnose issues without reading source code.

Definition of Done

• Prometheus metrics exposed on /metrics:
  • lldap_operator_reconcile_total{resource, result} — counter per CRD type and outcome (success, error, requeue).
  • lldap_operator_reconcile_duration_seconds{resource} — histogram of reconcile durations.
  • lldap_operator_lldap_api_requests_total{method, status} — counter of GraphQL/REST API calls.
  • lldap_operator_lldap_api_duration_seconds{method} — histogram.
  • lldap_operator_resources_managed{resource} — gauge of currently managed resources.
  • lldap_operator_queue_depth{resource} — gauge of pending reconciliations.
• Structured logging via tracing:
  • Every reconcile logs: resource name/namespace, action taken, duration.
  • Errors include full context (resource ref, lldap error, retry info).
  • Debug level logs individual API calls.
  • JSON format in production, human-readable in development.
• Kubernetes Events:
  • Normal/Created — resource created in lldap.
  • Normal/Updated — resource updated in lldap.
  • Normal/Deleted — resource deleted from lldap.
  • Normal/PasswordSet — password applied.
  • Warning/ReconcileError — reconciliation failed (with reason).
  • Warning/DependencyNotReady — waiting for dependency.
  • Warning/SecretNotFound — referenced Secret missing.
• Documentation: Document all available metrics with their labels and semantics in the docs. Document log levels and what information is available at each level. Add a Grafana dashboard JSON example to docs/examples/.

Technical Details

• Use prometheus crate with prometheus::Registry.
• Metrics collection: instrument the reconcile function and the lldap client.
• Event recorder: kube::runtime::events::Recorder stored in the Context.
• Log correlation: include the resource's uid and resourceVersion in tracing spans for log correlation.
• Avoid high-cardinality labels (no user IDs in metrics labels).