Splunk Operator: there is no way to upgrade from 9.3.x to 9.4.x

[yaroslav-nakonechnikov]

Please select the type of request

Bug

Tell us more

Describe the request
till now we are using splunk 9.3.8
all is good, all is working.

now we decided to upgrade to 9.4.8.
We replaced image for splunk-operator, and it started to work. But it fails to work on deployer:

TASK [splunk_common : Restrict permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key] ***
ok: [localhost]
Monday 16 February 2026  09:22:03 +0000 (0:00:00.811)       0:01:19.683 *******
FAILED - RETRYING: [localhost]: Start Splunk via CLI (5 retries left).
FAILED - RETRYING: [localhost]: Start Splunk via CLI (4 retries left).
FAILED - RETRYING: [localhost]: Start Splunk via CLI (3 retries left).
FAILED - RETRYING: [localhost]: Start Splunk via CLI (2 retries left).
FAILED - RETRYING: [localhost]: Start Splunk via CLI (1 retries left).

TASK [splunk_common : Start Splunk via CLI] ************************************
fatal: [localhost]: FAILED! => {
    "attempts": 5,
    "changed": false,
    "cmd": [
        "/opt/splunk/bin/splunk",
        "start",
        "--accept-license",
        "--answer-yes",
        "--no-prompt"
    ],
    "delta": "0:00:00.296681",
    "end": "2026-02-16 09:22:59.658520",
    "rc": 1,
    "start": "2026-02-16 09:22:59.361839"
}

STDOUT:


This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.


Perform migration and upgrade without previewing configuration changes? [y/n] y

Migrating to:
VERSION=9.4.8
BUILD=c543277b24fa
PRODUCT=splunk
PLATFORM=Linux-x86_64

Currently configured KVStore database path="/opt/splunk/var/lib/splunk/kvstore"

-> Currently used KVSTore version=5.0.32. Expected version=4.2 or version=7.0
-> isKVstoreDisabled=0
-> isKVstoreDatabaseFolderExist=1
-> isKVstoreDiagnosticsFolderExist=1
-> isKVstoreVersionFileFolderExist=1
-> isKVstoreVersionFileFolderEmpty=0
-> isKVstoreVersionFileMatched=1
-> isKVstoreVersionFromBsonMatched=0
* Active KVStore version upgrade precheck FAILED!
--
This check is to ensure that KVStore version 4.2 been in use.
--
In order to fix this failed check, re-install the previous Splunk version, and follow the KVStore upgrade documentation: https://docs.splunk.com/Documentation/Splunk/9.3.0/Admin/MigrateKVstore#Upgrade_KV_store_server_to_version_4.2 .

* CPU Info upgrade precheck PASSED
Some upgrade prechecks failed!


STDERR:


-- Migration information is being logged to '/opt/splunk/var/log/splunk/migration.log.2026-02-16.09-22-59' --
ERROR while running splunk-preinstall.


MSG:

non-zero return code

PLAY RECAP *********************************************************************
localhost                  : ok=105  changed=10   unreachable=0    failed=1    skipped=65   rescued=0    ignored=0

Monday 16 February 2026  09:22:59 +0000 (0:00:56.434)       0:02:16.117 *******
===============================================================================
splunk_common : Start Splunk via CLI ----------------------------------- 56.43s
splunk_common : Set options in saml ------------------------------------ 14.00s
splunk_common : Set options in role_admin ------------------------------- 3.24s
splunk_common : Set options in settings --------------------------------- 2.52s
splunk_common : Check if requests_unixsocket exists --------------------- 2.39s
Gathering Facts --------------------------------------------------------- 1.91s
Fetch adhoc playbooks --------------------------------------------------- 1.71s
splunk_common : Setup indexer discovery for index-clustering ------------ 1.69s
splunk_common : Set options in authentication --------------------------- 1.67s
splunk_common : Setup default tcpout group for index-clustering --------- 1.65s
splunk_common : Cleanup Splunk runtime files ---------------------------- 1.63s
splunk_common : Apply admin password ------------------------------------ 1.23s
splunk_common : Get Splunk status --------------------------------------- 1.16s
splunk_common : Get Splunk status --------------------------------------- 1.14s
splunk_common : Get Splunk status --------------------------------------- 1.12s
splunk_common : Set general pass4SymmKey -------------------------------- 1.04s
clean server.conf ------------------------------------------------------- 1.02s
Exec command to remove at_rbi_resmonitor from apps before boot. --------- 1.02s
splunk_common : Check for scloud ---------------------------------------- 0.99s
splunk_common : Find manifests ------------------------------------------ 0.98s

and i'm not sure how it actually worked on other pods.
We see that it is started, but it is started only in dev environment, where we don't have data. What will happen with KV storage at around 100Gb in size?

Expected behavior
it is expected to see clear steps how to upgrade splunk, which will not cause issues during upgrade.

Reproduction/Testing steps

• start cluster with 9.3.x
• try to upgrade it with replacing image for 9.4.x
• wait for result