OpenSearch: endpoint-separation guard runs after index bootstrap, so .os shadow indices are still created when ES==OS

[swicken]

Problem Statement

During QA of the ES→OpenSearch migration (surfaced by #36218, test case TC-037; part of epic #35476), the endpoint-separation guard fires but too late to be effective.

When the old engine (ES) and new engine (OS) are configured with the same address (DOT_ES_ENDPOINTS == DOT_OS_ENDPOINTS) and DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE=1, IndexStartupValidator correctly detects the overlap and halts the migration to Phase 0 — but it runs ~6 seconds after the index bootstrap has already created the .os shadow indices. As a result the .os indices and their indicies rows persist despite the migration being refused, failing TC-037's "no .os rows" acceptance criterion.

Root cause: ordering. IndexStartupValidator.validateIndexingConfig() runs after createContentIndexes() creates the shadow index set.

Impact: the separation guard's purpose (leave nothing behind when ES==OS) is not achieved; the halt only stops subsequent dual-write, not the initial .os creation. Migration is behind a feature flag, so production impact is limited to migration testing/rollout.

Steps to Reproduce

Rig: docker/docker-compose-examples/single-node-os-migration/ (old engine = OpenSearch 1.3.20 :9200, new engine = OpenSearch 3.4.0 :9201).

1. Point both engines at the same cluster and set Phase 1:

   DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE: '1'
   DOT_ES_ENDPOINTS: 'https://opensearch3:9200'   # == DOT_OS_ENDPOINTS
   DOT_ES_TLS_TRUST_SELF_SIGNED: 'true'

2. Bring up fresh (down -v then up -d); wait for dotCMS ready.
3. Check the DB: SELECT count(*) FROM indicies WHERE index_name LIKE '%.os';
4. Check the cluster: curl -sk .../9201/_cat/indices | grep '\.os'

Observed (guard fires, but .os created first):

15:01:44–45  OSIndexAPIImpl createIndex ...working_150142.os / ...live_150142.os   (phase still 1)
15:01:51     validation FAILED — "ES and OS clients point to the same endpoint(s): [opensearch3:9200]..."
15:01:51     Migration phase reset to PHASE_0_MIGRATION_NOT_STARTED

Result: 2 .os rows in indicies + 2 .os cluster indices (expected: 0).

Acceptance Criteria

• When DOT_ES_ENDPOINTS == DOT_OS_ENDPOINTS at startup with Phase ≥ 1, no .os indices are created and no .os rows exist in indicies.
• The separation guard runs before any index bootstrap/dual-write (or .os artifacts created before the halt are cleaned up on halt).
• The existing halt behavior (reset to Phase 0, ES-only fallback, "same endpoint(s)" log) is preserved.

dotCMS Version

dotcms/dotcms:latest (verified against release 26.06.30-01), single-node-os-migration rig, migration Phase 1.

Severity

Medium - Some functionality impacted

Links

NA — surfaced by QA of #36218 (epic #35476).

[github-actions]

PRs linked to this issue

• fix(opensearch): enforce ES/OS endpoint separation at the OS index-creation chokepoint (#36419) by @swicken