Upgrade ElasticSearch/OpenSearch Client Libraries

[pragya811]

Type of change

Note: Fill x in []

• bug
• enhancement
• documentation
• dependencies

Description

Migrate the cloud-governance application from elasticsearch-py 7.x + elasticsearch-dsl 7.x (talking to OpenSearch 1.x via compatibility mode) to a dual-client architecture:

• opensearch-py 3.2.0 for the Intlab OpenSearch v3 cluster (primary data store)
• elasticsearch-py 8.17.0 for the Ecoeng Elasticsearch 8.8.0 cluster (tenant-specific)

Key Changes:

Core library (cloud_governance/common/elasticsearch/elasticsearch_operations.py):

• Replace elasticsearch-dsl query builder with raw OpenSearch/ES query DSL
• Add ES_SERVER_TYPE environment variable (opensearch | elasticsearch) to select client
• Use opensearch-py client for OpenSearch clusters (with auth + SSL support)
• Use elasticsearch-py 8.x client for Elasticsearch clusters (no auth needed for Ecoeng)
• Remove deprecated doc_type parameter from all methods
• Fix post_query empty hits fallback from {} to []
• Add error logging to client initialization failures

Jenkins pipelines (all Jenkinsfiles + Python runners):

• Add ES_USER / ES_PASSWORD credential bindings for authenticated OpenSearch v3
• Pass es_user, es_password, and ES_SERVER_TYPE to all container invocations
• Set ES_SERVER_TYPE = 'elasticsearch' for Ecoeng tenants (ecoeng_01, _02, _03)

CI/CD (.github/workflows/):

• Replace elasticsearch:7.11.0 service container with opensearchproject/opensearch:3.2.0
• Add DISABLE_SECURITY_PLUGIN=true for test containers

Dependencies:

• Remove: elasticsearch-dsl==7.4.0
• Upgrade: elasticsearch 7.17.13 → 8.17.0
• Add: opensearch-py==3.2.0

Testing Performed

• Unit tests pass locally (all Python 3.10-3.14)
• Integration tests against local OpenSearch 3.2.0 container
• Jenkins validation: daily policies, cost explorer, CRO job against live Intlab OSv3
• Ecoeng validation: yearly_savings_report read+write against live Ecoeng ES 8.8.0
• Grafana data source connectivity confirmed against OSv3
• All index permissions verified on new cluster (including cloud_resource_orchestration_persistence_run and cloudability-cloud-governance-ibm-cost-usage-reports)

For security reasons, all pull requests need to be approved first before running any automated CI

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c6080eff-cf72-452b-a3a0-c4715ef1d991

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch es_upgrade

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}