Splunk Operator: CM and other pods not starting

[vinitmasaun]

Please select the type of request

Bug

Tell us more

Describe the request

• A clear and concise description of the request.
  Deployed sva c3 via helm with splunkoperator enabled set to true. The operator manager pod starts up successfully and goes into Ready state with no errors reported in the manager container logs. However, the CM pod keeps restarting and never goes into ready state with the following error reported in the splunkd.log:

02-25-2026 23:34:00.252 +0000 ERROR HttpClientRequest [1383 WebuiStartup] - HTTP client error=Connection refused while accessing server=http://127.0.0.1:8065 for request=http://127.0.0.1:8065/favicon.ico.
02-25-2026 23:34:00.302 +0000 WARN HttpClientRequest [1383 WebuiStartup] - Returning error HTTP/1.1 502 Connection refused
02-25-2026 23:34:00.302 +0000 ERROR HttpClientRequest [1316 TcpOutEloop] - Caught exception while parsing HTTP reply: Unexpected character while looking for value: '<'
02-25-2026 23:34:00.302 +0000 ERROR IndexerDiscoveryHeartbeatThread [1316 TcpOutEloop] - Error in Indexer Discovery communication. Verify that the pass4SymmKey set under [indexer_discovery:group1] in 'outputs.conf' matches the same setting under [indexer_discovery] in 'server.conf' on the cluster manager. [uri=https://localhost:8089/services/indexer_discovery http_code=502 http_response="Unauthorized"]
02-25-2026 23:34:00.512 +0000 ERROR HttpClientRequest [1383 WebuiStartup] - HTTP client error=Connection refused while accessing server=http://127.0.0.1:8065 for request=http://127.0.0.1:8065/favicon.ico.
02-25-2026 23:34:00.512 +0000 WARN HttpClientRequest [1383 WebuiStartup] - Returning error HTTP/1.1 502 Connection refused
02-25-2026 23:34:00.712 +0000 ERROR HttpClientRequest [1383 WebuiStartup] - HTTP client error=Connection refused while accessing server=http://127.0.0.1:8065 for request=http://127.0.0.1:8065/favicon.ico.
02-25-2026 23:34:00.732 +0000 WARN HttpClientRequest [1383 WebuiStartup] - Returning error HTTP/1.1 502 Connection refused
02-25-2026 23:34:00.732 +0000 ERROR ExecProcessor [1338 ExecProcessor] - message from "/opt/splunk/bin/python3.9 /opt/splunk/etc/apps/search/bin/quarantine_files.py" Unable to retrieve management host and port from filesystem; falling back to 'splunkd local-rest-uri'.
02-25-2026 23:34:00.732 +0000 ERROR ExecProcessor [1338 ExecProcessor] - message from "/opt/splunk/bin/python3.9 /opt/splunk/etc/apps/search/bin/quarantine_files.py" Traceback (most recent call last):
02-25-2026 23:34:00.732 +0000 ERROR ExecProcessor [1338 ExecProcessor] - message from "/opt/splunk/bin/python3.9 /opt/splunk/etc/apps/search/bin/quarantine_files.py" File "/opt/splunk/lib/python3.9/site-packages/splunk/clilib/cli_common.py", line 504, in getMgmtUri
02-25-2026 23:34:00.732 +0000 ERROR ExecProcessor [1338 ExecProcessor] - message from "/opt/splunk/bin/python3.9 /opt/splunk/etc/apps/search/bin/quarantine_files.py" if tmpProtocol in ['http', 'https'] and tmpHost and all(i in string.digits for i in tmpPort):
02-25-2026 23:34:00.732 +0000 ERROR ExecProcessor [1338 ExecProcessor] - message from "/opt/splunk/bin/python3.9 /opt/splunk/etc/apps/search/bin/quarantine_files.py" TypeError: 'int' object is not iterable
02-25-2026 23:34:00.932 +0000 ERROR HttpClientRequest [1383 WebuiStartup] - HTTP client error=Connection refused while accessing server=http://127.0.0.1:8065 for request=http://127.0.0.1:8065/favicon.ico.
02-25-2026 23:34:00.942 +0000 WARN HttpClientRequest [1383 WebuiStartup] - Returning error HTTP/1.1 502 Connection refused
02-25-2026 23:34:00.942 +0000 WARN SearchProcessRunner [1404 AuditSearchExecutor] - Preforked search process pool limits: max_search_process_pool=2048, manager_threads=1, enable_search_process_long_lifespan=1, max_search_process_per_manager=2048, max_idle_process_count=64, max_idle_process_memory=1048576, idle_process_hostwide_memory_threshold=90
02-25-2026 23:34:01.142 +0000 ERROR HttpClientRequest [1383 WebuiStartup] - HTTP client error=Connection refused while accessing server=http://127.0.0.1:8065 for request=http://127.0.0.1:8065/favicon.ico.
02-25-2026 23:34:01.142 +0000 WARN HttpClientRequest [1383 WebuiStartup] - Returning error HTTP/1.1 502 Connection refused
02-25-2026 23:34:01.382 +0000 ERROR HttpClientRequest [1383 WebuiStartup] - HTTP client error=Connection refused while accessing server=http://127.0.0.1:8065 for request=http://127.0.0.1:8065/favicon.ico.
02-25-2026 23:34:01.382 +0000 WARN HttpClientRequest [1383 WebuiStartup] - Returning error HTTP/1.1 502 Connection refused
command terminated with exit code 137

[Process exited with code 137]
Expected behavior

• A clear and concise description of what you expected to happen.
  The SVA c3 pods should all be created and go into running state.
  Splunk setup on K8S
• Details of the Splunk setup on the K8s cluster.

Reproduction/Testing steps

• Steps to reproduce the bug. For an enhancement or feature request, please provide steps to test.
  Deploy Splunk-enterprise helm chart on AWS EKS version v1.33.7 with automode.
  K8s environment
• Useful information about the K8S environment being used. Eg. version of K8s, kind of K8s cluster etc..
  AWS EKS version 1.33.7 with automode
  Proposed changes(optional)
• Proposed change, if any.

K8s collector data(optional)

• Please provide data collected from the K8s collectors, if any.

Additional context(optional)

• Add any other context about the problem here.