OpenDJ not starting with .bcfks keyfile

[mhixsoningenia]

I have a .bcfks keyfile that was created with a default password. Upon our product installation we need to change the keystore password to something new. I am using keytool to do so:

keytool -storetype BCFKS -providername BCFIPS -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath bc-fips-1.0.2.3.jar -keystore $TMP_DIR/certs/opendj-keystore.bcfks -storepasswd -storepass -new

This succeeds and I can use keytool to list the keys using the newly created password.

I am changing the keystore password before launching setup and installing OpenDJ.

However, start-ds successfully starts OpenDJ only if I don't change the default password on the keystore.

Here is a successful run, where the keystore password has not been change:

[18/Sep/2023:14:49:24 -0700] category=CORE severity=NOTICE msgID=134 msg=OpenDJ Server 4.5.6 (build 20230830085759, revision number 13fed9d) starting up
[18/Sep/2023:14:49:25 -0700] category=com.forgerock.opendj.ldap.core severity=NOTICE msgID=-1 msg=Attempting to install BC FIPS provider
[18/Sep/2023:14:49:25 -0700] category=com.forgerock.opendj.ldap.core severity=NOTICE msgID=-1 msg=BC Provider was registered already
[18/Sep/2023:14:49:25 -0700] category=JVM severity=NOTICE msgID=21 msg=Installation Directory: /opt/opendj
[18/Sep/2023:14:49:25 -0700] category=JVM severity=NOTICE msgID=23 msg=Instance Directory: /opt/opendj
[18/Sep/2023:14:49:25 -0700] category=JVM severity=NOTICE msgID=17 msg=JVM Information: 11.0.19+7-LTS by Red Hat, Inc., 64-bit architecture, 1166016512 bytes heap size
[18/Sep/2023:14:49:25 -0700] category=JVM severity=NOTICE msgID=18 msg=JVM Host: RHEL7, running Linux 3.10.0-1160.el7.x86_64 amd64, 4660940800 bytes physical memory size, number of processors available 4
[18/Sep/2023:14:49:25 -0700] category=JVM severity=NOTICE msgID=19 msg=JVM Arguments: "-Djava.security.properties=/opt/opendj/java.security.bouncycastle", "-Dorg.bouncycastle.fips.approved_only=true", "-Djava.security.properties=/opt/opendj/java.security.bouncycastle", "-Dorg.bouncycastle.fips.approved_only=true", "--add-exports=java.base/sun.security.x509=ALL-UNNAMED", "--add-exports=java.base/sun.security.tools.keytool=ALL-UNNAMED", "-Dorg.opends.server.scriptName=start-ds"
[18/Sep/2023:14:49:29 -0700] category=BACKEND severity=NOTICE msgID=513 msg=The database backend userRoot containing 0 entries has started
[18/Sep/2023:14:49:29 -0700] category=EXTENSIONS severity=NOTICE msgID=221 msg=DIGEST-MD5 SASL mechanism using a server fully qualified domain name of: localhost
[18/Sep/2023:14:49:30 -0700] category=PROTOCOL severity=ERROR msgID=1526 msg=The key with alias '[admin-cert]' was not found for 'Administration Connector'. Verify that the keystore is properly configured
[18/Sep/2023:14:49:30 -0700] category=PROTOCOL severity=WARNING msgID=1528 msg=Disabling Administration Connector
[18/Sep/2023:14:49:30 -0700] category=org.opends.messages.external severity=WARNING msgID=1 msg=Ignoring unsupported entry in 'jdk.tls.disabledAlgorithms': include jdk.disabled.namedCurves
[18/Sep/2023:14:49:30 -0700] category=org.opends.messages.external severity=WARNING msgID=1 msg=Ignoring unsupported entry in 'jdk.certpath.disabledAlgorithms': SHA1 jdkCA & usage TLSServer
[18/Sep/2023:14:49:30 -0700] category=org.opends.messages.external severity=WARNING msgID=1 msg=Ignoring unsupported entry in 'jdk.certpath.disabledAlgorithms': SHA1 usage SignedJAR & denyAfter 2019-01-01
[18/Sep/2023:14:49:30 -0700] category=org.opends.messages.external severity=WARNING msgID=1 msg=Ignoring unsupported entry in 'jdk.certpath.disabledAlgorithms': include jdk.disabled.namedCurves
[18/Sep/2023:14:49:31 -0700] category=CORE severity=NOTICE msgID=135 msg=The Directory Server has started successfully
[18/Sep/2023:14:49:31 -0700] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerStarted, alert ID org.opends.messages.core-135): The Directory Server has started successfully
[18/Sep/2023:14:49:31 -0700] category=PROTOCOL severity=NOTICE msgID=276 msg=Started listening for new connections on LDAPS Connection Handler 0.0.0.0 port 636
[18/Sep/2023:14:49:31 -0700] category=PROTOCOL severity=NOTICE msgID=276 msg=Started listening for new connections on LDAP Connection Handler 0.0.0.0 port 389

And here is a run after updating the keystore file with keytool.

[18/Sep/2023:14:58:50 -0700] category=CORE severity=NOTICE msgID=134 msg=OpenDJ Server 4.5.6 (build 20230830085759, revision number 13fed9d) starting up
[18/Sep/2023:14:58:51 -0700] category=com.forgerock.opendj.ldap.core severity=NOTICE msgID=-1 msg=Attempting to install BC FIPS provider
[18/Sep/2023:14:58:51 -0700] category=com.forgerock.opendj.ldap.core severity=NOTICE msgID=-1 msg=BC Provider was registered already
[18/Sep/2023:14:58:51 -0700] category=JVM severity=NOTICE msgID=21 msg=Installation Directory: /opt/opendj
[18/Sep/2023:14:58:51 -0700] category=JVM severity=NOTICE msgID=23 msg=Instance Directory: /opt/opendj
[18/Sep/2023:14:58:51 -0700] category=JVM severity=NOTICE msgID=17 msg=JVM Information: 11.0.19+7-LTS by Red Hat, Inc., 64-bit architecture, 1166016512 bytes heap size
[18/Sep/2023:14:58:51 -0700] category=JVM severity=NOTICE msgID=18 msg=JVM Host: RHEL7, running Linux 3.10.0-1160.el7.x86_64 amd64, 4660940800 bytes physical memory size, number of processors available 4
[18/Sep/2023:14:58:51 -0700] category=JVM severity=NOTICE msgID=19 msg=JVM Arguments: "-Djava.security.properties=/opt/opendj/java.security.bouncycastle", "-Dorg.bouncycastle.fips.approved_only=true", "-Djava.security.properties=/opt/opendj/java.security.bouncycastle", "-Dorg.bouncycastle.fips.approved_only=true", "--add-exports=java.base/sun.security.x509=ALL-UNNAMED", "--add-exports=java.base/sun.security.tools.keytool=ALL-UNNAMED", "-Dorg.opends.server.scriptName=start-ds"
[18/Sep/2023:14:58:55 -0700] category=BACKEND severity=NOTICE msgID=513 msg=The database backend userRoot containing 0 entries has started
[18/Sep/2023:14:58:55 -0700] category=EXTENSIONS severity=NOTICE msgID=221 msg=DIGEST-MD5 SASL mechanism using a server fully qualified domain name of: localhost
[18/Sep/2023:14:58:55 -0700] category=PROTOCOL severity=ERROR msgID=1526 msg=The key with alias '[admin-cert]' was not found for 'Administration Connector'. Verify that the keystore is properly configured
[18/Sep/2023:14:58:55 -0700] category=PROTOCOL severity=WARNING msgID=1528 msg=Disabling Administration Connector
[18/Sep/2023:14:58:55 -0700] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerShutdown, alert ID org.opends.messages.core-141): The Directory Server has started the shutdown process. The shutdown was initiated by an instance of class org.opends.server.core.DirectoryServer and the reason provided for the shutdown was An error occurred while trying to start the Directory Server: InitializationException: An error occurred while attempting to initialize the SSL context for use in the LDAP Connection Handler: An error occurred while trying to create a key manager factory to access the contents of keystore file /opt/opendj/certs/opendj-keystore.bcfks: UnrecoverableKeyException(BCFKS KeyStore unable to recover private key (asdf): Error finalising cipher data: mac check in CCM failed) (id=org.opends.messages.extension-83) (LDAPConnectionHandler2.java:491 AdministrationConnector.java:137 ConnectionHandlerConfigManager.java:262 ConnectionHandlerConfigManager.java:221 DirectoryServer.java:1756 DirectoryServer.java:1486 DirectoryServer.java:5047)
[18/Sep/2023:14:58:55 -0700] category=BACKEND severity=NOTICE msgID=370 msg=The backend userRoot is now taken offline
[18/Sep/2023:14:58:55 -0700] category=CORE severity=NOTICE msgID=203 msg=The Directory Server is now stopped

In both runs there is an error message containing in part: "ERROR msgID=1526 msg=The key with alias '[admin-cert]' ".
I've never had a key with that alias and it doesn't seem to be a problem since it runs correct in the first case.

Only in the second, failed run is my alias of "asdf" mentioned.

Here is keytool listing the key in question, with the correct alias, after the password had been changed:

keytool -list -keystore /opt/opendj/certs/opendj-keystore.bcfks -storetype BCFKS -providername BCFIPS -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /tmp/bc-fips-1.0.2.3.jar -storepass Ytd35dVc6KbkAt8
Keystore type: BCFKS
Keystore provider: BCFIPS

Your keystore contains 1 entry

asdf, Aug 17, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): A6:[REDACTED]:49

Is there any reason why OpenDJ doesn't run with my BCFKS keystore after I've used keytool to modify the password?

java --version
openjdk 11.0.19 2023-04-18 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.19.0.7-1.el7_9) (build 11.0.19+7-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.19.0.7-1.el7_9) (build 11.0.19+7-LTS, mixed mode, sharing)

[vharseko]

@maximthomas

[maximthomas]

@mhixsoningenia hi, could you provide /opt/opendj/logs/errors log file?

[maximthomas]

does password form keystore.pin matches your keystore?

keytool -list -keystore /opt/opendj/certs/opendj-keystore.bcfks -storetype BCFKS -providername BCFIPS -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /tmp/bc-fips-1.0.2.3.jar -storepass:file /opt/opendj/config/keystore.pin

[mhixsoningenia]

Yes.

keytool -list -keystore /opt/opendj/certs/opendj-keystore.bcfks -storetype BCFKS -providername BCFIPS -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /tmp/bc-fips-1.0.2.3.jar -storepass:file /opt/opendj/config/keystore.pin
Keystore type: BCFKS
Keystore provider: BCFIPS

Your keystore contains 1 entry

asdf, Aug 17, 2023, PrivateKeyEntry,
Certificate fingerprint (SHA-256): A6:BF:2A:45[REDACTED]B:F1:4A:FE:23:68:49

[maximthomas]

There should be two entries: admin-cert and server-cert in the keystore. Please check the following page:
https://github.com/OpenIdentityPlatform/OpenDJ/wiki/How-to-Setup-OpenDJ-with-BCFKS-FIPS-Key-Store-Type-support

[mhixsoningenia]

The strange thing is that I've never added any key to the keystore with either alias of 'admin-cert' nor 'server-cert' but OpenDJ works with the opendj-keystore.bcfks before I change the password.

[mhixsoningenia]

As a side note, in all of my research into getting OpenDJ to work with BCFKS keystores I've never come across the wiki page you linked to. In fact, Google exactly that page's title and Google doesn't know about it.

Just seems strange that its not more easily findable, nor linked to from other sources like Stack Overflow.