OKV platform certificate rotation - pitfall , awareness!!!!

OKV version: 21.9

Setup: Multimaster R/W cluster

Plan:

https://docs.google.com/spreadsheets/d/e/2PACX-1vSaXXTjj9cE1fvYpNmsDNBOkTIw78yTwQ6a9oGkowba0E3sTqiny_KLfrzM-RYMSQ/pubhtml

Screenshots:

https://docs.google.com/document/d/e/2PACX-1vTLvcW6cBytYzxFMeMMHauXYJLJF1z9_jHkq1ocqa9SkJ92GVVRy-J6nFUOHYu4Tg/pub

Issues faced:

Post redo shipping platform certificate were rotated, we had platform instability coming from non sync password file between node 1 and node 2 oracle db.

Fix:

1. Manually sync the password file from node 1 to node 2

2. restart both nodes

Concern: Node , platform certificate rotation operations are online operation. But the redo shipping operation landing us in downtime for endpoint is a concern!!!!!!!!!!!!!!!!!!!!!!!!

Thanks

 

Solution Test: Can a temporary JAVA_HOME facilitate endpoint certificate rotation in OKV

Environment: 

1. Standalone Oracle DB (TDE key is stored in OKV)

2. Multimaster OKV (2 nodes – 1 subgroup and 1 group) cluster.

Current Setup Status: Works (refer to previous logs & video)

Current Setup config:

1. JAVA_HOME comes from oracle binary

2. ORACLE_HOME is available

3. DB is online

Prereqs to test the solution:

1. Rename the ORACLE_HOME, so okvutil doesn’t refer to the JAVA from ORACLE_HOME (note $OKV_HOME/conf/okvclient.ora still has JAVA_HOME reference).

2. Download a portable version of JDK version 8 & Unpack it in a permanent location

3. Reference that for JAVA_HOME and ensure the okvutil does refer to the new path rather than old ORACLE_HOME when launched

4. Validate the current certificate status in OKV console

5. Rotate the certificate, it takes few mins

6. Collect the new certificate status in OKV console

7. Use the temporary JAVA_HOME and fetch the certificate using okvutil

8. Document the observation

1. Rename the ORACLE_HOME, so okvutil doesn’t refer to the JAVA from ORACLE_HOME (note $OKV_HOME/conf/okvclient.ora still has JAVA_HOME reference).

[oracle@vcentos79-oracle-ggtgt ~]$ cd $ORACLE_HOME [oracle@vcentos79-oracle-ggtgt db_1]$ pwd /u01/app/oracle/product/19.0.0/db_1 [oracle@vcentos79-oracle-ggtgt db_1]$ cd .. [oracle@vcentos79-oracle-ggtgt 19.0.0]$ ls -altr total 4 drwxr-xr-x.  3 oracle oinstall   20 Jul 16  2023 .. drwxr-xr-x.  3 oracle oinstall   18 Jul 16  2023 . drwxr-xr-x. 72 oracle oinstall 4096 Nov  2  2024 db_1 [oracle@vcentos79-oracle-ggtgt 19.0.0]$ ps -ef|grep -i db_1 oracle   19685 11236  0 14:24 pts/0    00:00:00 grep --color=auto -i db_1 [oracle@vcentos79-oracle-ggtgt 19.0.0]$ fuser -c db_1 /u01/app/oracle/product/19.0.0/db_1: 11236c [oracle@vcentos79-oracle-ggtgt 19.0.0]$ ps -ef|grep 11236 oracle   11236 11235  0 11:58 pts/0    00:00:00 -bash oracle   19708 11236  0 14:24 pts/0    00:00:00 ps -ef oracle   19709 11236  0 14:24 pts/0    00:00:00 grep --color=auto 11236 [oracle@vcentos79-oracle-ggtgt 19.0.0]$ cd [oracle@vcentos79-oracle-ggtgt ~]$ fuser -c /u01/app/oracle/product/19.0.0/db_1 [oracle@vcentos79-oracle-ggtgt ~]$ mv /u01/app/oracle/product/19.0.0/db_1 /u01/app/oracle/product/19.0.0/db_1_test [oracle@vcentos79-oracle-ggtgt ~]$

2. Download a portable version of JDK version 8 & Unpack it in a permanent location

cd /home/oracle/dba

mkdir TEMP_JAVA_BASE

cd TEMP_JAVA_BASE

wget https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u462-b08/OpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz

mkdir -p java-1.8

tar -zxvf OpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz -C ./java-1.8

[oracle@vcentos79-oracle-ggtgt dba]$ mkdir TEMP_JAVA_BASE [oracle@vcentos79-oracle-ggtgt dba]$ cd TEMP_JAVA_BASE/ [oracle@vcentos79-oracle-ggtgt TEMP_JAVA_BASE]$ ls -altr total 0 drwxr-xr-x. 9 oracle oinstall 225 Jun 22 13:27 .. drwxr-xr-x. 2 oracle oinstall   6 Jun 22 13:27 . [oracle@vcentos79-oracle-ggtgt TEMP_JAVA_BASE]$ wget https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u462-b08/OpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz --2026-06-22 13:27:53--  https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u462-b08/OpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz Resolving github.com (github.com)... 20.26.156.215 Connecting to github.com (github.com)|20.26.156.215|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://release-assets.githubusercontent.com/github-production-release-asset/372924428/248229d4-4ada-4efb-a430-53dfb5693776?sp=r&sv=2018-11-09&sr=b&spr=https&se=2026-06-22T13%3A22%3A28Z&rscd=attachment%3B+filename%3DOpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz&rsct=application%2Foctet-stream&skoid=96c2d410-5711-43a1-aedd-ab1947aa7ab0&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skt=2026-06-22T12%3A21%3A34Z&ske=2026-06-22T13%3A22%3A28Z&sks=b&skv=2018-11-09&sig=gEMmnzRHnNoE6bncypsFo4ms74qI6MZZv4aUIol93Yo%3D&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmVsZWFzZS1hc3NldHMuZ2l0aHVidXNlcmNvbnRlbnQuY29tIiwia2V5Ijoia2V5MSIsImV4cCI6MTc4MjEzMzEwMSwibmJmIjoxNzgyMTMxMzAxLCJwYXRoIjoicmVsZWFzZWFzc2V0cHJvZHVjdGlvbi5ibG9iLmNvcmUud2luZG93cy5uZXQifQ.x-Qvdl4IrRKkoHc2noUYtC19t0MEJOucS8bGTMWrA80&response-content-disposition=attachment%3B%20filename%3DOpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz&response-content-type=application%2Foctet-stream [following] --2026-06-22 13:27:53--  https://release-assets.githubusercontent.com/github-production-release-asset/372924428/248229d4-4ada-4efb-a430-53dfb5693776?sp=r&sv=2018-11-09&sr=b&spr=https&se=2026-06-22T13%3A22%3A28Z&rscd=attachment%3B+filename%3DOpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz&rsct=application%2Foctet-stream&skoid=96c2d410-5711-43a1-aedd-ab1947aa7ab0&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skt=2026-06-22T12%3A21%3A34Z&ske=2026-06-22T13%3A22%3A28Z&sks=b&skv=2018-11-09&sig=gEMmnzRHnNoE6bncypsFo4ms74qI6MZZv4aUIol93Yo%3D&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmVsZWFzZS1hc3NldHMuZ2l0aHVidXNlcmNvbnRlbnQuY29tIiwia2V5Ijoia2V5MSIsImV4cCI6MTc4MjEzMzEwMSwibmJmIjoxNzgyMTMxMzAxLCJwYXRoIjoicmVsZWFzZWFzc2V0cHJvZHVjdGlvbi5ibG9iLmNvcmUud2luZG93cy5uZXQifQ.x-Qvdl4IrRKkoHc2noUYtC19t0MEJOucS8bGTMWrA80&response-content-disposition=attachment%3B%20filename%3DOpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz&response-content-type=application%2Foctet-stream Resolving release-assets.githubusercontent.com (release-assets.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ... Connecting to release-assets.githubusercontent.com (release-assets.githubusercontent.com)|185.199.109.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 103087414 (98M) [application/octet-stream] Saving to: ‘OpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz’ 100%[================================================================================>] 103,087,414 4.40MB/s   in 22s 2026-06-22 13:28:15 (4.46 MB/s) - ‘OpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz’ saved [103087414/103087414] [oracle@vcentos79-oracle-ggtgt TEMP_JAVA_BASE]$ ls -altr total 100672 -rw-r--r--. 1 oracle oinstall 103087414 Jul 23  2025 OpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz drwxr-xr-x. 9 oracle oinstall       225 Jun 22 13:27 .. drwxr-xr-x. 2 oracle oinstall        61 Jun 22 13:27 . [oracle@vcentos79-oracle-ggtgt TEMP_JAVA_BASE]$ mkdir java-1.8 [oracle@vcentos79-oracle-ggtgt TEMP_JAVA_BASE]$ tar -zxvf OpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz -C ./java-1.8 jdk8u462-b08/ jdk8u462-b08/include/ jdk8u462-b08/include/jni.h jdk8u462-b08/include/jvmti.h jdk8u462-b08/include/classfile_constants.h jdk8u462-b08/include/linux/ jdk8u462-b08/include/linux/jni_md.h jdk8u462-b08/include/linux/jawt_md.h jdk8u462-b08/include/jvmticmlr.h jdk8u462-b08/include/jawt.h jdk8u462-b08/include/jdwpTransport.h … jdk8u462-b08/sample/jmx/jmx-scandir/build.properties jdk8u462-b08/sample/jmx/jmx-scandir/src/ jdk8u462-b08/sample/jmx/jmx-scandir/src/com/ jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/ jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/ jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/ jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/ jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/ScanManagerMXBean.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/package.html jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/ScanDirConfigMXBean.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/ResultLogManager.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/ScanDirConfig.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/DirectoryScannerMXBean.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/DirectoryScanner.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/ResultLogManagerMXBean.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/ScanManager.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/ScanDirClient.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/config/ jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/config/XmlConfigUtils.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/config/package.html jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/config/ResultLogConfig.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/config/ResultRecord.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/config/ScanManagerConfig.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/config/DirectoryScannerConfig.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/config/FileMatch.java jdk8u462-b08/sample/jmx/jmx-scandir/src/com/sun/jmx/examples/scandir/ScanDirAgent.java jdk8u462-b08/sample/jmx/jmx-scandir/src/etc/ jdk8u462-b08/sample/jmx/jmx-scandir/src/etc/password.properties jdk8u462-b08/sample/jmx/jmx-scandir/src/etc/testconfig.xml jdk8u462-b08/sample/jmx/jmx-scandir/src/etc/management.properties jdk8u462-b08/sample/jmx/jmx-scandir/src/etc/access.properties jdk8u462-b08/sample/jmx/jmx-scandir/index.html jdk8u462-b08/sample/nbproject/ jdk8u462-b08/sample/nbproject/project.xml jdk8u462-b08/sample/nio/ .. jdk8u462-b08/sample/nio/file/ jdk8u462-b08/sample/nio/file/WatchDir.java jdk8u462-b08/sample/nio/file/Chmod.java jdk8u462-b08/sample/nio/file/DiskUsage.java jdk8u462-b08/sample/nio/file/Xdd.java jdk8u462-b08/sample/nio/file/FileType.java jdk8u462-b08/sample/nio/file/Copy.java jdk8u462-b08/sample/nio/file/AclEdit.java jdk8u462-b08/sample/try-with-resources/ jdk8u462-b08/sample/try-with-resources/src/ jdk8u462-b08/sample/try-with-resources/src/Unzip.java jdk8u462-b08/sample/try-with-resources/src/CustomAutoCloseableSample.java jdk8u462-b08/sample/try-with-resources/src/ZipCat.java jdk8u462-b08/sample/try-with-resources/index.html jdk8u462-b08/sample/lambda/ jdk8u462-b08/sample/lambda/BulkDataOperations/ jdk8u462-b08/sample/lambda/BulkDataOperations/src/ jdk8u462-b08/sample/lambda/BulkDataOperations/src/WC.java jdk8u462-b08/sample/lambda/BulkDataOperations/src/PasswordGenerator.java jdk8u462-b08/sample/lambda/BulkDataOperations/src/CSVProcessor.java jdk8u462-b08/sample/lambda/BulkDataOperations/src/Grep.java jdk8u462-b08/sample/lambda/BulkDataOperations/index.html jdk8u462-b08/sample/lambda/DefaultMethods/ jdk8u462-b08/sample/lambda/DefaultMethods/Inheritance.java jdk8u462-b08/sample/lambda/DefaultMethods/SimplestUsage.java jdk8u462-b08/sample/lambda/DefaultMethods/MixIn.java jdk8u462-b08/sample/lambda/DefaultMethods/DiamondInheritance.java jdk8u462-b08/sample/lambda/DefaultMethods/ArrayIterator.java jdk8u462-b08/sample/lambda/DefaultMethods/Reflection.java jdk8u462-b08/sample/scripting/ jdk8u462-b08/sample/scripting/scriptpad/ jdk8u462-b08/sample/scripting/scriptpad/README.txt jdk8u462-b08/sample/scripting/scriptpad/nbproject/ jdk8u462-b08/sample/scripting/scriptpad/nbproject/file-targets.xml jdk8u462-b08/sample/scripting/scriptpad/nbproject/project.xml jdk8u462-b08/sample/scripting/scriptpad/nbproject/netbeans-targets.xml jdk8u462-b08/sample/scripting/scriptpad/nbproject/jdk.xml jdk8u462-b08/sample/scripting/scriptpad/build.xml jdk8u462-b08/sample/scripting/scriptpad/build.properties jdk8u462-b08/sample/scripting/scriptpad/src/ jdk8u462-b08/sample/scripting/scriptpad/src/com/ jdk8u462-b08/sample/scripting/scriptpad/src/com/sun/ jdk8u462-b08/sample/scripting/scriptpad/src/com/sun/sample/ jdk8u462-b08/sample/scripting/scriptpad/src/com/sun/sample/scriptpad/ jdk8u462-b08/sample/scripting/scriptpad/src/com/sun/sample/scriptpad/Main.java jdk8u462-b08/sample/scripting/scriptpad/src/META-INF/ jdk8u462-b08/sample/scripting/scriptpad/src/META-INF/manifest.mf jdk8u462-b08/sample/scripting/scriptpad/src/scripts/ jdk8u462-b08/sample/scripting/scriptpad/src/scripts/textcolor.js jdk8u462-b08/sample/scripting/scriptpad/src/scripts/README.txt jdk8u462-b08/sample/scripting/scriptpad/src/scripts/memory.sh jdk8u462-b08/sample/scripting/scriptpad/src/scripts/memory.js jdk8u462-b08/sample/scripting/scriptpad/src/scripts/mail.js jdk8u462-b08/sample/scripting/scriptpad/src/scripts/insertfile.js jdk8u462-b08/sample/scripting/scriptpad/src/scripts/linewrap.js jdk8u462-b08/sample/scripting/scriptpad/src/scripts/browse.js jdk8u462-b08/sample/scripting/scriptpad/src/scripts/memory.bat jdk8u462-b08/sample/scripting/scriptpad/src/scripts/memmonitor.js jdk8u462-b08/sample/scripting/scriptpad/src/resources/ jdk8u462-b08/sample/scripting/scriptpad/src/resources/mm.js jdk8u462-b08/sample/scripting/scriptpad/src/resources/scriptpad.js jdk8u462-b08/sample/scripting/scriptpad/src/resources/conc.js jdk8u462-b08/sample/scripting/scriptpad/src/resources/gui.js jdk8u462-b08/sample/scripting/scriptpad/src/resources/Main.js jdk8u462-b08/jre/ jdk8u462-b08/jre/LICENSE jdk8u462-b08/jre/THIRD_PARTY_README jdk8u462-b08/jre/ASSEMBLY_EXCEPTION jdk8u462-b08/jre/lib/ jdk8u462-b08/jre/lib/rt.jar jdk8u462-b08/jre/lib/jexec jdk8u462-b08/jre/lib/psfontj2d.properties jdk8u462-b08/jre/lib/net.properties jdk8u462-b08/jre/lib/charsets.jar … jdk8u462-b08/jre/bin/unpack200 jdk8u462-b08/jre/bin/policytool jdk8u462-b08/jre/bin/java jdk8u462-b08/release [oracle@vcentos79-oracle-ggtgt TEMP_JAVA_BASE]$ pwd /home/oracle/dba/TEMP_JAVA_BASE [oracle@vcentos79-oracle-ggtgt TEMP_JAVA_BASE]$ ls -altr total 100672 -rw-r--r--. 1 oracle oinstall 103087414 Jul 23  2025 OpenJDK8U-jdk_x64_linux_hotspot_8u462b08.tar.gz drwxr-xr-x. 9 oracle oinstall       225 Jun 22 13:27 .. drwxr-xr-x. 3 oracle oinstall        77 Jun 22 13:28 . drwxr-xr-x. 3 oracle oinstall        26 Jun 22 13:28 java-1.8 [oracle@vcentos79-oracle-ggtgt TEMP_JAVA_BASE]$ cd java-1.8/ [oracle@vcentos79-oracle-ggtgt java-1.8]$ ls -altr total 0 drwxr-xr-x. 8 oracle oinstall 190 Jul 16  2025 jdk8u462-b08 drwxr-xr-x. 3 oracle oinstall  77 Jun 22 13:28 .. drwxr-xr-x. 3 oracle oinstall  26 Jun 22 13:28 . [oracle@vcentos79-oracle-ggtgt java-1.8]$ which java /usr/bin/which: no java in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/oracle/.local/bin:/home/oracle/bin:/u01/app/oracle/product/19.0.0/db_1/bin) [oracle@vcentos79-oracle-ggtgt java-1.8]$ [oracle@vcentos79-oracle-ggtgt java-1.8]$ [oracle@vcentos79-oracle-ggtgt java-1.8]$ pwd /home/oracle/dba/TEMP_JAVA_BASE/java-1.8 [oracle@vcentos79-oracle-ggtgt java-1.8]$ ls -altr total 0 drwxr-xr-x. 8 oracle oinstall 190 Jul 16  2025 jdk8u462-b08 drwxr-xr-x. 3 oracle oinstall  77 Jun 22 13:28 .. drwxr-xr-x. 3 oracle oinstall  26 Jun 22 13:28 . [oracle@vcentos79-oracle-ggtgt java-1.8]$ cd jdk8u462-b08/ [oracle@vcentos79-oracle-ggtgt jdk8u462-b08]$ ls -altr total 51608 drwxr-xr-x.  3 oracle oinstall      144 Jul 16  2025 lib drwxr-xr-x.  4 oracle oinstall       47 Jul 16  2025 man -r--r--r--.  1 oracle oinstall   158248 Jul 16  2025 THIRD_PARTY_README drwxr-xr-x. 10 oracle oinstall      151 Jul 16  2025 sample -r--r--r--.  1 oracle oinstall    19274 Jul 16  2025 LICENSE drwxr-xr-x.  4 oracle oinstall       95 Jul 16  2025 jre drwxr-xr-x.  3 oracle oinstall      132 Jul 16  2025 include -r--r--r--.  1 oracle oinstall     1522 Jul 16  2025 ASSEMBLY_EXCEPTION -rw-r--r--.  1 oracle oinstall 52649665 Jul 16  2025 src.zip -rw-r--r--.  1 oracle oinstall      483 Jul 16  2025 release -rw-r--r--.  1 oracle oinstall     2400 Jul 16  2025 NOTICE drwxr-xr-x.  2 oracle oinstall     4096 Jul 16  2025 bin drwxr-xr-x.  8 oracle oinstall      190 Jul 16  2025 . drwxr-xr-x.  3 oracle oinstall       26 Jun 22 13:28 .. [oracle@vcentos79-oracle-ggtgt jdk8u462-b08]$ pwd /home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08 [oracle@vcentos79-oracle-ggtgt jdk8u462-b08]$ export JAVA_HOME=/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08 [oracle@vcentos79-oracle-ggtgt jdk8u462-b08]$ export PATH=$PATH:$JAVA_HOME:$JAVA_HOME/bin [oracle@vcentos79-oracle-ggtgt jdk8u462-b08]$ java -version openjdk version "1.8.0_462" OpenJDK Runtime Environment (Temurin)(build 1.8.0_462-b08) OpenJDK 64-Bit Server VM (Temurin)(build 25.462-b08, mixed mode) [oracle@vcentos79-oracle-ggtgt jdk8u462-b08]$ which java ~/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/bin/java [oracle@vcentos79-oracle-ggtgt jdk8u462-b08]$

3. Reference that for JAVA_HOME and ensure the okvutil does refer to the new path rather than old ORACLE_HOME when launched

1. Verify first the okvutil fails not finding the java bin..

[oracle@vcentos79-oracle-ggtgt ~]$ $OKV_HOME/bin/okvutil list The JAVA_HOME property specified in the conf/okvclient.ora must point to a valid JDK/JRE installation. [oracle@vcentos79-oracle-ggtgt ~]$

Note.. before ORACLE_HOME rename the okvutil referenced the JAVA from ORACLE_HOME using okvclient.ora. Strace evidence…

[oracle@vcentos79-oracle-ggtgt ~]$ grep -i java strace_output.txt 19479 open("/u01/app/oracle/product/19.0.0/db_1/jdk/jre/lib/amd64/libjava.so", O_RDONLY) = 3</u01/app/oracle/product/19.0.0/db_1/jdk/jre/lib/amd64/libjava.so> 19479 open("/u01/app/oracle/product/19.0.0/db_1/jdk/jre/lib/amd64/libjava.so", O_RDONLY|O_CLOEXEC) = 3</u01/app/oracle/product/19.0.0/db_1/jdk/jre/lib/amd64/libjava.so> 19479 openat(AT_FDCWD, "/usr/java/packages/lib/ext", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory) 19490 open("/u01/app/oracle/product/19.0.0/db_1/jdk/jre/lib/amd64/libjava.so", O_RDONLY) = 3</u01/app/oracle/product/19.0.0/db_1/jdk/jre/lib/amd64/libjava.so> 19490 open("/u01/app/oracle/product/19.0.0/db_1/jdk/jre/lib/amd64/libjava.so", O_RDONLY|O_CLOEXEC) = 3</u01/app/oracle/product/19.0.0/db_1/jdk/jre/lib/amd64/libjava.so> 19490 openat(AT_FDCWD, "/usr/java/packages/lib/ext", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory) [oracle@vcentos79-oracle-ggtgt ~]$ [oracle@vcentos79-oracle-ggtgt ~]$ cat $OKV_HOME/conf/okvclient.ora SERVER=192.168.194.122:5696/CN=server_cert,OU=Key_Vault,O=Oracle,L=Redwood_City,ST=California,C=us,192.168.194.123:5696/CN=node_2_cert,OU=Key_Vault,O=Oracle,L=Redwood_City,ST=California,C=us CONF_ID=XkkPNeQLIdice2pe SERVER_DN=CN=server_cert,OU=Key_Vault,O=Oracle,L=Redwood_City,ST=California,C=us GEN_TIMESTAMP=2026-06-19 03\:16\:45 UTC UPDATE_TIMESTAMP=2026-06-22 12:34:32 GMT SW_TYPE=ENROLLED_ENDPOINT_SOFTWARE JAVA_HOME=/u01/app/oracle/product/19.0.0/db_1/jdk OKV_JVM_LIB_PATH=/u01/app/oracle/product/19.0.0/db_1/jdk/jre/lib/amd64/server/libjvm.so EP_TYPE=Oracle Database OKV_HOSTNAME=voracle8x-oracle-key-vault2 SERVER_POLL_TIMEOUT=300.00 SSL_WALLET_LOC=/u01/app/oracle/admin/GGTGT06T/wallet_062026/okv/ssl _NOT_STRICT_PKCS11=1 PKCS11_NO_KMIP_OBJECT_ACCESS_CHECK=0 PKCS11_CACHE_TIMEOUT=60.00 PKCS11_PERSISTENT_CACHE_TIMEOUT=1440.00 PKCS11_PERSISTENT_CACHE_FIRST=1 PKCS11_PERSISTENT_CACHE_REFRESH_WINDOW=30.00 PKCS11_CONFIG_PARAM_REFRESH_INTERVAL=10.00 _TRACE_DIR=. _TRACE_LEVEL=0 NUM_AFFINITY_RW_NODES=2 NUM_AFFINITY_RO_NODES=0 FIPS_CONFIG=DEFAULT [oracle@vcentos79-oracle-ggtgt ~]$

2. Reference the new JAVA_HOME and fetch the certificate

[oracle@vcentos79-oracle-ggtgt ~]$ export JAVA_HOME=/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08 [oracle@vcentos79-oracle-ggtgt ~]$ export PATH=$PATH:$JAVA_HOME/bin [oracle@vcentos79-oracle-ggtgt ~]$ export OKV_HOME=/u01/app/oracle/admin/GGTGT06T/wallet_062026/okv [oracle@vcentos79-oracle-ggtgt ~]$ $OKV_HOME/bin/okvutil list Enter Oracle Key Vault endpoint password: Unique ID                               Type            Identifier 757EA53F-9257-4BA1-BE1E-2F4872E6FB0E    Symmetric Key   TDE Master Encryption Key: MKID 06DE6C421CA03D4F65BFD58A4A2E6D931E D99FEB87-C5AD-45AA-B661-E4BB7FAA4AB8    Template        Default template for ENDPT_VCENTOS79-ORACLE-GGTGT_GGTGT06T [oracle@vcentos79-oracle-ggtgt ~]$

Strace output confirms the new JAVA_HOME usage…

[oracle@vcentos79-oracle-ggtgt ~]$ grep -i java strace_output_newjvm.txt|grep so|grep -v ENOENT 20061 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/bin/../lib/amd64/jli/libjli.so", O_RDONLY|O_CLOEXEC) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/lib/amd64/jli/libjli.so> 20061 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/server/libjvm.so", O_RDONLY|O_CLOEXEC) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/server/libjvm.so> 20062 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libverify.so", O_RDONLY) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libverify.so> 20062 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libverify.so", O_RDONLY|O_CLOEXEC) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libverify.so> 20062 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libjava.so", O_RDONLY) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libjava.so> 20062 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libjava.so", O_RDONLY|O_CLOEXEC) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libjava.so> 20062 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so", O_RDONLY) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so> 20062 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so", O_RDONLY|O_CLOEXEC) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so> 20062 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so", O_RDONLY) = 4</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so> 20073 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/bin/../lib/amd64/jli/libjli.so", O_RDONLY|O_CLOEXEC) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/lib/amd64/jli/libjli.so> 20073 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/server/libjvm.so", O_RDONLY|O_CLOEXEC) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/server/libjvm.so> 20074 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libverify.so", O_RDONLY) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libverify.so> 20074 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libverify.so", O_RDONLY|O_CLOEXEC) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libverify.so> 20074 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libjava.so", O_RDONLY) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libjava.so> 20074 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libjava.so", O_RDONLY|O_CLOEXEC) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libjava.so> 20074 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so", O_RDONLY) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so> 20074 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so", O_RDONLY|O_CLOEXEC) = 3</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so> 20074 open("/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so", O_RDONLY) = 4</home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08/jre/lib/amd64/libzip.so> [oracle@vcentos79-oracle-ggtgt ~]$

3. Validate the current certificate status in OKV console

Expiry date: 19 june 2027

4. Rotate the certificate, it takes few mins & Collect the new certificate status in OKV console

Certificate Rotation in Progress | One or more endpoints rotated successfully.

new date: 20-JUN-2027

CA changed to: Updating to current certificate issuer

5. Use the temporary JAVA_HOME and fetch the certificate using okvutil

export JAVA_HOME=/home/oracle/dba/TEMP_JAVA_BASE/java-1.8/jdk8u462-b08

export PATH=$PATH:$JAVA_HOME/bin

export OKV_HOME=/u01/app/oracle/admin/GGTGT06T/wallet_062026/okv

$OKV_HOME/bin/okvutil list

Attempt 1	[oracle@vcentos79-oracle-ggtgt ~]$ $OKV_HOME/bin/okvutil list Enter Oracle Key Vault endpoint password: Unique ID                               Type            Identifier 757EA53F-9257-4BA1-BE1E-2F4872E6FB0E    Symmetric Key   TDE Master Encryption Key: MKID 06DE6C421CA03D4F65BFD58A4A2E6D931E D99FEB87-C5AD-45AA-B661-E4BB7FAA4AB8    Template        Default template for ENDPT_VCENTOS79-ORACLE-GGTGT_GGTGT06T [oracle@vcentos79-oracle-ggtgt ~]$
Attempt 2	[oracle@vcentos79-oracle-ggtgt ~]$ $OKV_HOME/bin/okvutil list -v 4 okvutil version 21.9.0.0.0 Endpoint type: Oracle Database Configuration file: /u01/app/oracle/admin/GGTGT06T/wallet_062026/okv/conf/okvclient.ora Server: 192.168.194.123:5696 192.168.194.122:5696 Standby Servers: No auto-login wallet found, password needed Enter Oracle Key Vault endpoint password: Trying to connect to 192.168.194.123:5696 ... Trying to connect to 192.168.194.122:5696 ... Trying to connect to 192.168.194.123:5696 ... Trying to connect to 192.168.194.122:5696 ... Attempt to connect to 192.168.194.123 failed with: 29106 Incorrect password supplied. Attempt to connect to 192.168.194.122 failed with: 29106 Incorrect password supplied. Error: Server Connect Failed [oracle@vcentos79-oracle-ggtgt ~]$
Attempt 3	[oracle@vcentos79-oracle-ggtgt ~]$ $OKV_HOME/bin/okvutil list Enter Oracle Key Vault endpoint password: Error: Server Connect Failed [oracle@vcentos79-oracle-ggtgt ~]$

We can see new certificate any way as we see below…

[oracle@vcentos79-oracle-ggtgt ~]$ ls -altr $OKV_HOME/ssl

total 8

drwxr-xr-x. 9 oracle oinstall   86 Jun 19 04:28 ..

-rw-------. 1 oracle oinstall 4361 Jun 19 04:28 ewallet.p12

drwxr-x---. 3 oracle oinstall   53 Jun 22 14:37 .

drwxr-x---. 2 oracle oinstall   25 Jun 22 14:37 19700101000000000000

[oracle@vcentos79-oracle-ggtgt ~]$ ls -altr $OKV_HOME/ssl/19700101000000000000

total 8

drwxr-x---. 3 oracle oinstall   53 Jun 22 14:37 ..

-rw-------. 1 oracle oinstall 4361 Jun 22 14:37 ewallet.p12

drwxr-x---. 2 oracle oinstall   25 Jun 22 14:37 .

[oracle@vcentos79-oracle-ggtgt ~]$

Appendix A: Restoring ORACLE_HOME doesn’t fix the issue…

[oracle@vcentos79-oracle-ggtgt ~]$ mv /u01/app/oracle/product/19.0.0/db_1_test/ /u01/app/oracle/product/19.0.0/db_1 [oracle@vcentos79-oracle-ggtgt ~]$ . oraenv ORACLE_SID = [oracle] ? GGTGT06T The Oracle base has been set to /u01/app/oracle [oracle@vcentos79-oracle-ggtgt ~]$ export OKV_HOME=/u01/app/oracle/admin/GGTGT06T/wallet_062026/okv [oracle@vcentos79-oracle-ggtgt ~]$ ls -altr pro* -rwxr-xr-x. 1 oracle oinstall 218 Jun 19 03:06 profile_GGTGT06T.bkp -rwxr-xr-x. 1 oracle oinstall 225 Jun 19 04:22 profile_GGTGT06T [oracle@vcentos79-oracle-ggtgt ~]$ . ./profile_GGTGT06T [oracle@vcentos79-oracle-ggtgt ~]$ env|grep ORA ORACLE_SID=GGTGT06T ORACLE_BASE=/u01/app/oracle ORAENV_ASK=NO ORACLE_HOME=/u01/app/oracle/product/19.0.0/db_1 [oracle@vcentos79-oracle-ggtgt ~]$ env|grep OKV OKV_HOME=/u01/app/oracle/admin/GGTGT06T/wallet_062026/okv [oracle@vcentos79-oracle-ggtgt ~]$ [oracle@vcentos79-oracle-ggtgt ~]$ [oracle@vcentos79-oracle-ggtgt ~]$ export JAVA_HOME=$ORACLE_HOME/jdk [oracle@vcentos79-oracle-ggtgt ~]$ export PATH=$PATH:$JAVA_HOME/bin [oracle@vcentos79-oracle-ggtgt ~]$ java -version java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode) [oracle@vcentos79-oracle-ggtgt ~]$ which java /u01/app/oracle/product/19.0.0/db_1/jdk/bin/java [oracle@vcentos79-oracle-ggtgt ~]$ $OKV_HOME/bin/okvutil list Enter Oracle Key Vault endpoint password: Error: Server Connect Failed [oracle@vcentos79-oracle-ggtgt ~]$ [oracle@vcentos79-oracle-ggtgt ~]$ [oracle@vcentos79-oracle-ggtgt ~]$ $OKV_HOME/bin/okvutil list -v 4 okvutil version 21.9.0.0.0 Endpoint type: Oracle Database Configuration file: /u01/app/oracle/admin/GGTGT06T/wallet_062026/okv/conf/okvclient.ora Server: 192.168.194.123:5696 192.168.194.122:5696 Standby Servers: No auto-login wallet found, password needed Enter Oracle Key Vault endpoint password: Trying to connect to 192.168.194.123:5696 ... Trying to connect to 192.168.194.122:5696 ... Trying to connect to 192.168.194.123:5696 ... Trying to connect to 192.168.194.122:5696 ... Attempt to connect to 192.168.194.123 failed with: 29106 Incorrect password supplied. Attempt to connect to 192.168.194.122 failed with: 29106 Incorrect password supplied. Error: Server Connect Failed [oracle@vcentos79-oracle-ggtgt ~]$

The only option now left is to reenroll the Endpoint.

Thanks