[QA-G6] OpenSearch Migration — Index Delete: ES vs OS Behavior (TC-016–TC-020)

[fabrizzio-dotCMS]

Parent Epic

#35476

Glossary (quick)

• Old engine (ES) — Elasticsearch, what dotCMS uses today. Port 9200, inspected with Kibana (5601).
• New engine (OS) — OpenSearch, the migration target. Port 9201, inspected with OpenSearch Dashboards (5602).
• Migration phase — 0 = old engine only · 1 = write to both, search the old one · 2 = write to both, read OS.
• Dual-write — in Phase 1 every content change is written to both engines.
• .os suffix (the "tag") — marks a new-engine index. New-engine index names end in .os (e.g. cluster_08abc3.working_20260406.os); old-engine names do not. Visible everywhere: cluster, DB, REST. (NOT an os:: prefix — that's an obsolete model.)

Environment

docker compose -f docker/docker-compose-examples/os-migration/docker-compose.yml up -d

Service	URL	Used for
Old engine (ES 7.10)	http://localhost:9200	Direct REST queries
Kibana	http://localhost:5601	Inspect/query the OLD index
New engine (OS 3.x)	http://localhost:9201	Direct REST queries
OpenSearch Dashboards	http://localhost:5602	Inspect/query the NEW index
dotCMS	http://localhost:8082	Admin UI + REST API (admin:admin)

Migration settings live in dotmarketing-config.properties (FEATURE_FLAG_OPEN_SEARCH_PHASE, OS_ENDPOINTS=http://localhost:9201, OS_AUTH_TYPE=BASIC, OS_AUTH_BASIC_USER=admin, OS_AUTH_BASIC_PASSWORD=admin, OS_TLS_ENABLED=false).

Helpers (referenced by the steps below)

H1 — Find your real index names. Names contain a timestamp (differ per machine). New-engine indexes end in .os; old-engine names do not.

curl -s -u admin:admin http://localhost:8082/api/v1/esindex | jq .   # what dotCMS knows (keeps .os, drops cluster prefix)
curl -s -u admin:admin http://localhost:9200/_cat/indices?v          # OLD engine physical names (no .os)
curl -s -u admin:admin http://localhost:9201/_cat/indices?v          # NEW engine physical names (end in .os)

Or Admin → System → Index.

H3 — Check the database (SQL console; new-engine marker is the .os suffix, not an os:: prefix):

SELECT * FROM indicies WHERE index_name LIKE '%.os';      -- NEW engine rows
SELECT * FROM indicies WHERE index_name NOT LIKE '%.os';  -- OLD engine rows
SELECT CASE WHEN index_name LIKE '%.os' THEN 'NEW' ELSE 'OLD' END AS engine, COUNT(*) FROM indicies GROUP BY 1;

H4 — Count documents in an index (use full physical names from H1; new-engine one ends in .os):

curl -s -u admin:admin http://localhost:9200/<old-index-name>/_count      # e.g. cluster_X.working_20230101
curl -s -u admin:admin http://localhost:9201/<new-index-name.os>/_count    # e.g. cluster_X.working_20260406.os

Test Cases

TC-016 — Deleting the old-engine index does not touch the new-engine index

⚠️ Open design question (confirm before pass/fail). The two engines hold the same logical index, distinguished only by the .os suffix. It is not settled whether deleting "an index" should remove only the named one (what the doc implies — "User-triggered index shutdown → not replicated to OS") or cascade to the .os counterpart. Run this once to observe what the endpoint actually does, then confirm with the team. The expected result below reflects the documented design; flag any divergence rather than failing.

• Objective: With differently-named indexes, observe whether deleting the old-engine index leaves the new-engine index intact (documented) or also removes it.
• Risk: High
• Preconditions: Divergent-name situation: start in Phase 0, create content (old-engine index gets an older timestamp), then switch to Phase 1 (new-engine index created later → newer timestamp + .os; e.g. old cluster_X.working_20230101 vs new cluster_X.working_20260406.os).
• Steps:
  1. Confirm names differ (only the new one ends in .os): SELECT index_name FROM indicies WHERE index_name LIKE '%working%';
  2. Delete the old-engine index (no .os): curl -s -u admin@dotcms.com:admin -X DELETE http://localhost:8082/api/v1/esindex/<old-index-name>
  3. Old engine gone? curl -s -u admin:admin http://localhost:9200/<old-index-name> → 404.
  4. New engine still there? curl -s -u admin:admin http://localhost:9201/<new-index-name.os> → 200.
  5. Re-run the SQL from step 1.
• Expected Result (documented design):
  • Delete returns HTTP 200; old-engine index physically deleted (404); new-engine index still exists (200); old-engine row removed from indicies, the .os row unchanged. (If both are deleted, record it for the team — see open question.)

TC-017 — Deleting an index that doesn't exist returns a clean 404

• Objective: A non-existent name returns a readable 404 (not a stack trace) and changes nothing.
• Risk: Medium
• Preconditions: dotCMS in any phase.
• Steps:
  1. curl -s -i -u admin@dotcms.com:admin -X DELETE http://localhost:8082/api/v1/esindex/working_totally_nonexistent_20990101
  2. Run H3 before and after to confirm no change.
• Expected Result: HTTP 404; readable error body (no raw stack trace); indicies unchanged.

TC-018 — Deleting the currently active index is rejected

• Objective: The active working index is in use; deletion must be refused.
• Risk: High
• Preconditions: dotCMS in Phase 0.
• Steps:
  1. Get active working index: curl -s -u admin:admin http://localhost:8082/api/v1/esindex/active/working
  2. Attempt delete: curl -s -i -u admin@dotcms.com:admin -X DELETE http://localhost:8082/api/v1/esindex/<active-index-name>
• Expected Result: HTTP 4xx (400/403); response explains the active index can't be deleted; index still exists; indicies row unchanged.

TC-019 — Deleting a non-active index from the admin UI works end-to-end

• Objective: A non-active index deleted via UI disappears from UI, engine, and DB — no error shown.
• Risk: Medium
• Preconditions: dotCMS in Phase 1.
• Steps:
  1. Create a spare index: curl -s -u admin@dotcms.com:admin -X PUT http://localhost:8082/api/v1/esindex/create/shards/1. Note the name.
  2. Admin → System → Index, find it, click Delete, confirm.
  3. Kibana: GET /<new-index-name> → 404.
  4. SQL: SELECT * FROM indicies WHERE index_name = '<deleted_name>';
• Expected Result: index gone from UI list; physically deleted (404); row gone from indicies; no error toast; no unhandled exception in log.

TC-020 — A full rebuild in Phase 1 rebuilds the old engine but preserves the new-engine index

• Objective: A full reindex creates a fresh old-engine index but must not recreate/wipe the new-engine index; dual-write resumes afterward.
• Risk: High
• Preconditions: dotCMS in Phase 1 with divergent index names (old without .os; new with a different timestamp + .os).
• Steps:
  1. Note the new-engine index name (.os) and its doc count (H1, H4).
  2. Admin → System → Index → Reindex. Wait for completion.
  3. In Kibana, identify the now-active old-engine index.
  4. In OpenSearch Dashboards, confirm the new-engine index is the same as before.
  5. Run H3.
  6. After rebuild, publish a new content item; check it landed in both engines (H4 / search).
  7. Search the content UI for pre-rebuild items.
• Expected Result:
  • New old-engine index (fresh timestamp) created and activated; previous old-engine index deactivated/removed after swap.
  • New-engine index is the same as before — same name, same doc count.
  • New content after rebuild goes to the new old-engine index and the existing new-engine index (dual-write resumed).
  • indicies shows the new old-engine row and the same .os row.
  • Pre-rebuild content still searchable.

Acceptance Criteria

• TC-016: delete old-engine index → 200; old gone (404); new (.os) intact (200); old row removed, .os row unchanged. (Record if both deleted — open question.)
• TC-017: delete non-existent → 404, readable body, indicies unchanged.
• TC-018: delete active index → 4xx; index & row unchanged.
• TC-019: delete non-active via UI → gone from UI, engine (404), and DB; no error.
• TC-020: full reindex rebuilds ES (new index activated) while the .os index is preserved unchanged; dual-write resumes.

Notes

• TC-016 cascade-vs-no-cascade is an OPEN design question — observe and report, don't assume. (See also Group 15 TC-048/049 for reindex-to-both, decision B.)

[KevinDavilaDotCMS]

QA Test Results — #35640

TC-016 — Deleting the old-engine index does not touch the new-engine index

Status: ⚠️ Partially Passed — Issues Found

• The old engine index was deleted successfully (returns 404 from Elasticsearch endpoint).
• Issue: The .os index was also removed even though I only called the endpoint to remove the ES index, not the .os one.
• Issue (maybe, to confirm): Neither index was deleted from the indices table.
• Issue (maybe, to confirm): The DELETE api/v1/esindex/${name} endpoint does not recognize the full index name.
  • Full DB name: cluster_kdavila-opensearch-2506-1.live_20260625201916.os
  • Name recognized by endpoint: live_20260625201916.os
  • Using the full name returns 404.

---

TC-017 — Deleting an index that doesn't exist returns a clean 404

Status: ✅ Passed

Note: Same index naming issue as TC-016 applies here.

---

TC-018 — Deleting the currently active index is rejected

Status: ❌ Failed — Unexpected Behavior

• The endpoint api/v1/esindex/active/working is deprecated and does not work.
• I listed all indexes and only one was present: working_xxxx (the live index was already deleted in TC-016).
• I ran the delete endpoint against it and it succeeded — the active index was deleted.
• Result: The instance now has zero indexes. This is likely not the expected behavior; the endpoint should reject deletion of the currently active index.

---

TC-019 — Deleting a non-active index from the admin UI works end-to-end

Status: ✅ Passed (with caveat)

• The endpoint /api/v1/esindex/create/shards/1 is deprecated and does not work.
• I proceeded with the test using available endpoints.
• The index was successfully removed from both the UI and the database.

[jcastro-dotcms]

NOTE TO DEVS

@fabrizzio-dotCMS one more thing to consider is that we should pop up a confirmation message when deleting an Index. This is a destruction action and there's no way to roll it back. A confirmation message could definitely prevent accidental deletion of important Indexes.