[Feature] Structured JSON logging with debug mode

[Siddhant-K-code]

Summary

Add structured JSON logging with a --debug flag that shows detailed pipeline internals.

Motivation

• Production deployments need machine-parseable logs for aggregation (ELK, Loki, Datadog)
• Debug mode helps users understand clustering decisions and tune parameters
• Trace IDs in logs connect to OpenTelemetry spans

Design

Normal mode (default)

{"level":"info","ts":"2026-02-15T10:00:00Z","msg":"request completed","path":"/v1/dedupe","input":12,"output":8,"clusters":8,"latency_ms":14,"trace_id":"abc123"}

Debug mode (--debug or --log-level debug)

{"level":"debug","ts":"...","msg":"clustering","input_count":12,"threshold":0.15,"clusters_formed":8,"largest_cluster":3,"smallest_cluster":1}
{"level":"debug","ts":"...","msg":"cluster_detail","cluster_id":0,"members":["chunk_1","chunk_5","chunk_9"],"centroid_distance":0.08}
{"level":"debug","ts":"...","msg":"selection","strategy":"score","cluster_id":0,"selected":"chunk_1","score":0.95}
{"level":"debug","ts":"...","msg":"mmr_iteration","step":1,"selected":"chunk_3","relevance":0.91,"diversity":0.87,"mmr_score":0.89}

Implementation

Use log/slog (stdlib, Go 1.21+):

// pkg/logging/logging.go
func Init(level string, format string) *slog.Logger

• --log-level: debug, info, warn, error (default: info)
• --log-format: json, text (default: json)
• --debug: shorthand for --log-level debug --log-format text

Deliverables

• pkg/logging/logging.go - Logger initialization
• Add --log-level, --log-format, --debug flags to root command
• Instrument cmd/api.go and cmd/serve.go with structured logs
• Add trace_id to all request logs when tracing is enabled
• Config file support under logging: section

Acceptance Criteria

• Default output is JSON, one line per log entry
• --debug shows clustering decisions and MMR iterations
• Trace IDs appear in logs when OTEL is enabled
• No performance impact when debug is off