While upgrading a Splunk Enterprise server from 9.3.x to 9.4.12, the RPM installation failed during the pre-upgrade checks. The error pointed to KV Store: Currently used KVStore version=4.0.24-linux-splunk-v1 Expected version=4.2 or version=7 Active KVStore version upgrade precheck FAILED! Some upgrade prechecks failed! pre install check failed My first assumption was that I simply needed to upgrade the KV Store version. That turned out to be only part of the story. The actual problem was that KV Store was not healthy in the first place. Once I started troubleshooting that, I found two separate issues: an expired Splunk server certificate and incorrect permissions on the KV Store key file. Here is how I worked through it. Checking the KV Store status I started by checking the KV Store status: $SPLUNK_HOME/bin/splunk show kvstore-status --verbose The important part of the output was: status : failed storageEngine : wiredTiger There was also an error while trying to retrieve...
Scribbled by Jithin Alex.