While upgrading a Splunk Enterprise server from 9.3.x to 9.4.12, the RPM installation failed during the pre-upgrade checks.
The error pointed to KV Store:
Currently used KVStore version=4.0.24-linux-splunk-v1
Expected version=4.2 or version=7
Active KVStore version upgrade precheck FAILED!
Some upgrade prechecks failed!
pre install check failed
My first assumption was that I simply needed to upgrade the KV Store version.
That turned out to be only part of the story.
The actual problem was that KV Store was not healthy in the first place. Once I started troubleshooting that, I found two separate issues: an expired Splunk server certificate and incorrect permissions on the KV Store key file.
Here is how I worked through it.
Checking the KV Store status
I started by checking the KV Store status:
$SPLUNK_HOME/bin/splunk show kvstore-status --verbose
The important part of the output was:
status : failed
storageEngine : wiredTiger
There was also an error while trying to retrieve the feature compatibility version.
At this point, I decided not to attempt any KV Store migration. If KV Store itself was not starting, upgrading it was unlikely to solve the actual problem.
The next place to look was mongod.log.
tail -100 $SPLUNK_HOME/var/log/splunk/mongod.log
That immediately revealed something useful:
The provided SSL certificate is expired or not yet valid.
Fatal Assertion 28652
aborting after fassert() failure
So MongoDB, which Splunk uses for KV Store, was refusing to start because of a certificate problem.
Finding the certificate Splunk was using
I checked the effective Splunk configuration with btool:
$SPLUNK_HOME/bin/splunk cmd btool server list sslConfig --debug
The server was using the standard Splunk certificate paths:
caCertFile = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem
I then checked the validity of server.pem:
$SPLUNK_HOME/bin/splunk cmd openssl x509 \
-in $SPLUNK_HOME/etc/auth/server.pem \
-noout -subject -issuer -dates
The result explained the problem:
notBefore=Nov 27 13:16:00 2019 GMT
notAfter=Nov 26 13:16:00 2022 GMT
The certificate had expired years earlier.
Interestingly, Splunk itself had continued running, so the issue only became obvious when the KV Store process needed to start properly during the upgrade activity.
I also checked the CA certificate separately. The CA was still valid, so there was no reason to touch it.
Replacing the expired server certificate
Before making any changes, I backed up the existing certificate.
After stopping Splunk, I copied the old certificate and moved it out of the active path:
sudo cp -p \
$SPLUNK_HOME/etc/auth/server.pem \
$SPLUNK_HOME/etc/auth/server.pem.expired_backup
sudo mv \
$SPLUNK_HOME/etc/auth/server.pem \
$SPLUNK_HOME/etc/auth/server.pem.old
I then started Splunk again.
Since this environment was using Splunk's default certificate setup, Splunk generated a new server.pem.
I checked the new certificate:
$SPLUNK_HOME/bin/splunk cmd openssl x509 \
-in $SPLUNK_HOME/etc/auth/server.pem \
-noout -subject -issuer -dates
The replacement certificate was valid until 2029.
So the certificate issue was fixed.
But KV Store still wasn't coming up.
The second problem
After another restart, show kvstore-status still showed:
status : failed
This time the error was more specific:
No suitable servers found
failed to connect to target host: 127.0.0.1:8191
That meant nothing was listening on the KV Store port.
I went back to mongod.log and checked the latest entries.
This time the error was:
permissions on
$SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key
are too open
So after fixing the certificate, MongoDB was getting further into the startup process but was now refusing to use the KV Store key file because its filesystem permissions were too permissive.
I fixed the ownership and permissions:
sudo chown splunk:splunk \
$SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key
sudo chmod 600 \
$SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key
The resulting permissions looked like:
-rw------- 1 splunk splunk ... splunk.key
I restarted Splunk once again.
KV Store finally came back
After the restart, I checked:
$SPLUNK_HOME/bin/splunk show kvstore-status --verbose
This time the result was what I wanted:
featureCompatibilityVersion : 4.2
status : ready
storageEngine : wiredTiger
serverVersion : 4.2.17
KV Store was healthy again.
More importantly, the server was already running KV Store version 4.2.17, which satisfied the prerequisite for the Splunk 9.4 upgrade.
So there was no need to manually migrate KV Store at that point.
What actually happened
The upgrade error initially looked like a simple KV Store version mismatch.
In reality, the chain of events was:
Splunk upgrade precheck failed
↓
KV Store status was failed
↓
mongod.log showed expired certificate
↓
server.pem was replaced
↓
KV Store still failed
↓
mongod.log showed splunk.key permissions too open
↓
Permissions corrected
↓
KV Store started successfully
↓
KV Store 4.2.17 confirmed
There were therefore two separate problems:
1. The default Splunk server certificate had expired.
MongoDB refused to start because server.pem was no longer valid.
2. The KV Store key file had overly permissive permissions.
After the certificate was fixed, MongoDB refused to start until splunk.key was restricted to the Splunk account.
A few useful commands
Check KV Store status:
$SPLUNK_HOME/bin/splunk show kvstore-status --verbose
Check the MongoDB log:
tail -100 $SPLUNK_HOME/var/log/splunk/mongod.log
Check the effective SSL configuration:
$SPLUNK_HOME/bin/splunk cmd btool server list sslConfig --debug
Check a certificate's validity:
$SPLUNK_HOME/bin/splunk cmd openssl x509 \
-in $SPLUNK_HOME/etc/auth/server.pem \
-noout -subject -issuer -dates
Check whether MongoDB is running:
ps -ef | grep [m]ongod
Check whether KV Store is listening on port 8191:
ss -lntp | grep 8191
What I would avoid
One thing I would not do immediately in this situation is:
splunk clean kvstore
A failed KV Store does not automatically mean that the database is corrupted.
In this case, the KV Store data itself was fine. MongoDB simply could not start because of an expired certificate and incorrect file permissions.
I would also avoid modifying files under:
$SPLUNK_HOME/etc/system/default/
unless there is a very specific reason to do so.
Final takeaway
The most useful lesson from this issue was not to take the upgrade precheck message too literally.
The installer complained about the KV Store version, but the underlying issue was that KV Store was unhealthy.
Checking these two things first saved a lot of unnecessary troubleshooting:
splunk show kvstore-status --verbose
and:
$SPLUNK_HOME/var/log/splunk/mongod.log
Once the certificate and key-file permissions were corrected, KV Store returned to ready, version 4.2.17 was detected correctly, and the server was ready to continue with the Splunk 9.4 upgrade.