wandb: raise init_timeout, add retry, fix shared-mode init for cross-region clusters

[DavidBellamy]

Summary

Fix WANDB_MODE=online booting past the 90s init timeout on cross-region clusters. Unblocks LLM360/RL360#243.

Root cause correction

Issue LLM360/RL360#243's original hypothesis ("primary↔secondary local socket handshake race") is incorrect. Per the wandb upstream feature PR (wandb/wandb#6882):

Make it possible to log to the same run using multiple writers with independent wandb-core's (including ones running, for example, on different machines)

Shared mode spawns one wandb-core per writer; each writer talks to wandb cloud independently; server-side aggregation by run_id. No local socket handshake.

The observed >90s timeout is just wandb's default init_timeout = 90.0s. The cross-continent HTTPS round-trip (MBZUAI Abu Dhabi ↔ wandb cloud US-West) with concurrent actor bursts can exceed 90s when the wandb cloud is under any load.

Changes

• init_timeout=300.0 on both primary and secondary Settings (configurable via WANDB_INIT_TIMEOUT_SECS)
• _wandb_init_with_retry wraps both init paths, 3 attempts, 5→10→20s exponential backoff on CommError/UsageError (configurable)
• x_label per-rank tagging per wandb distributed-training docs
• Drop reinit=True from secondary init_kwargs (not needed for shared mode, triggered stale-state warnings)

Unlocks

• WANDB_MODE=offline can be removed from scale.yaml.extra_env default
• ~/bin/wandb-sync-rl360.sh tmux patchwork on David's M2 account becomes obsolete
• Near-realtime dashboards replace 2-min-lag offline sync
• Per-rank system metrics via x_label filtering in wandb UI

Test plan

1. Merge → octopus → rebuild agentic-rl-nightly-YYYYMMDD image (same path as the TITO fix landed today)
2. Pilot run with a test composition that forces WANDB_MODE=online
3. Verify boot completes <5 min, primary+secondaries attach cleanly, dashboards live
4. Stress: 3 concurrent pilot-scale jobs (~100 concurrent wandb clients across them) vs wandb's documented 300-concurrent-client limit
5. Rollback: if anything misbehaves, env-var override WANDB_INIT_TIMEOUT_SECS=90 restores prior default in-place

Upstream

Will also open the equivalent PR against radixark/miles:main for upstream hygiene. Landing here first to unblock the M2 pilot via the LLM360-deploy octopus.