Second ejb remote call fails with EJBCLIENT000079: Unable to discover destination for request for EJB StatelessEJBLocator for "ejbca/cesecore-ejb/CertificateProfileSessionBean"

[franck-mosse]

Hello,
I am trying to perform an ejb remote call to EJBCA 7.11.0 from a java standard client (openjdk 11).
The calls are using SSL and client authenticates with user/password.
My objective is to perform several calls to ejbca to retrieve the id of an existing certificate profile.
For testing purpose my source code does twice the same call using the same reference to a remote ejb bean.

All works fine for the first call but a second call fails with the following error ( see my logs just after):

Initial context done
Ejb remote lookup done
Example result: 572782751

Error accessing remote bean
jakarta.ejb.NoSuchEJBException: EJBCLIENT000079: Unable to discover destination for request for EJB StatelessEJBLocator for "ejbca/cesecore-ejb/CertificateProfileSessionBean", view is interface org.cesecore.certificates.certificateprofile.CertificateProfileSessionRemote, affinity is Cluster "ejb"
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:620)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocationResult(RemotingEJBClientInterceptor.java:57)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocationResult(TransactionPostDiscoveryInterceptor.java:148)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocationResult(DiscoveryEJBClientInterceptor.java:130)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocationResult(NamingEJBClientInterceptor.java:87)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.call(AuthenticationContextEJBClientInterceptor.java:59)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.handleInvocationResult(AuthenticationContextEJBClientInterceptor.java:52)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.TransactionInterceptor.handleInvocationResult(TransactionInterceptor.java:212)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:1003)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:182)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:116)
at com.sun.proxy.$Proxy0.getCertificateProfileId(Unknown Source)
at Main.run3(Main.java:209)
at Main.main(Main.java:222)
Suppressed: java.net.ConnectException: Connection refused: no further information
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at org.xnio.nio.WorkerThread$ConnectHandle.handleReady(WorkerThread.java:327)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
at ...asynchronous invocation...(Unknown Source)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:599)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:565)
at org.jboss.remoting3.ConnectionInfo$None.getConnection(ConnectionInfo.java:82)
at org.jboss.remoting3.ConnectionInfo.getConnection(ConnectionInfo.java:55)
at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:499)
at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:445)
at org.jboss.remoting3.UncloseableEndpoint.getConnectedIdentity(UncloseableEndpoint.java:52)
at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider.getConnectedIdentityUsingClusterEffective(RemotingEJBDiscoveryProvider.java:342)
at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt.connectAndDiscover(RemotingEJBDiscoveryProvider.java:436)
at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider.discover(RemotingEJBDiscoveryProvider.java:207)
at org.jboss.ejb.protocol.remote.RemoteEJBDiscoveryConfigurator.lambda$configure$0(RemoteEJBDiscoveryConfigurator.java:42)
at org.wildfly.discovery.ConfiguredProvider.discover(ConfiguredProvider.java:47)
at org.wildfly.discovery.Discovery.discover(Discovery.java:122)
at org.wildfly.discovery.Discovery.discover(Discovery.java:93)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.discover(DiscoveryEJBClientInterceptor.java:346)
at org.wildfly.common.context.Contextual.runBiFunction(Contextual.java:224)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.discover(DiscoveryEJBClientInterceptor.java:341)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.doClusterDiscovery(DiscoveryEJBClientInterceptor.java:636)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.executeDiscovery(DiscoveryEJBClientInterceptor.java:419)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:110)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:514)
at org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:69)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:514)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.lambda$handleInvocation$0(AuthenticationContextEJBClientInterceptor.java:45)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.call(AuthenticationContextEJBClientInterceptor.java:59)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.handleInvocation(AuthenticationContextEJBClientInterceptor.java:44)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:514)
at org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:205)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:514)
at org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:341)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:178)
... 4 more
Exception in thread "main" java.lang.RuntimeException: jakarta.ejb.NoSuchEJBException: EJBCLIENT000079: Unable to discover destination for request for EJB StatelessEJBLocator for "ejbca/cesecore-ejb/CertificateProfileSessionBean", view is interface org.cesecore.certificates.certificateprofile.CertificateProfileSessionRemote, affinity is Cluster "ejb"
at Main.run3(Main.java:214)
at Main.main(Main.java:222)
Caused by: jakarta.ejb.NoSuchEJBException: EJBCLIENT000079: Unable to discover destination for request for EJB StatelessEJBLocator for "ejbca/cesecore-ejb/CertificateProfileSessionBean", view is interface org.cesecore.certificates.certificateprofile.CertificateProfileSessionRemote, affinity is Cluster "ejb"
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:620)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocationResult(RemotingEJBClientInterceptor.java:57)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocationResult(TransactionPostDiscoveryInterceptor.java:148)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocationResult(DiscoveryEJBClientInterceptor.java:130)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocationResult(NamingEJBClientInterceptor.java:87)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.call(AuthenticationContextEJBClientInterceptor.java:59)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.handleInvocationResult(AuthenticationContextEJBClientInterceptor.java:52)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.TransactionInterceptor.handleInvocationResult(TransactionInterceptor.java:212)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:1003)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:182)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:116)
at com.sun.proxy.$Proxy0.getCertificateProfileId(Unknown Source)
at Main.run3(Main.java:209)
... 1 more
Suppressed: java.net.ConnectException: Connection refused: no further information
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at org.xnio.nio.WorkerThread$ConnectHandle.handleReady(WorkerThread.java:327)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
at ...asynchronous invocation...(Unknown Source)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:599)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:565)
at org.jboss.remoting3.ConnectionInfo$None.getConnection(ConnectionInfo.java:82)
at org.jboss.remoting3.ConnectionInfo.getConnection(ConnectionInfo.java:55)
at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:499)
at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:445)
at org.jboss.remoting3.UncloseableEndpoint.getConnectedIdentity(UncloseableEndpoint.java:52)
at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider.getConnectedIdentityUsingClusterEffective(RemotingEJBDiscoveryProvider.java:342)
at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt.connectAndDiscover(RemotingEJBDiscoveryProvider.java:436)
at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider.discover(RemotingEJBDiscoveryProvider.java:207)
at org.jboss.ejb.protocol.remote.RemoteEJBDiscoveryConfigurator.lambda$configure$0(RemoteEJBDiscoveryConfigurator.java:42)
at org.wildfly.discovery.ConfiguredProvider.discover(ConfiguredProvider.java:47)
at org.wildfly.discovery.Discovery.discover(Discovery.java:122)
at org.wildfly.discovery.Discovery.discover(Discovery.java:93)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.discover(DiscoveryEJBClientInterceptor.java:346)
at org.wildfly.common.context.Contextual.runBiFunction(Contextual.java:224)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.discover(DiscoveryEJBClientInterceptor.java:341)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.doClusterDiscovery(DiscoveryEJBClientInterceptor.java:636)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.executeDiscovery(DiscoveryEJBClientInterceptor.java:419)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:110)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:514)
at org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:69)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:514)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.lambda$handleInvocation$0(AuthenticationContextEJBClientInterceptor.java:45)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.call(AuthenticationContextEJBClientInterceptor.java:59)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.handleInvocation(AuthenticationContextEJBClientInterceptor.java:44)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:514)
at org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:205)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:514)
at org.wildfly.common.context.Contextual.runExConsumer(Contextual.java:203)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequestInitial(EJBClientInvocationContext.java:341)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:178)
... 4 more

Apparently I can not reuse the same object reference for successive calls. If I perform a lookup before each call I don’t see the error but performance are drastically reduced.

Do you have an idea to solve correctly this issue without impacting the performance.

My environment is like here under :
• Centos 7
• OpenJDK 11 (64-it)
• Wildfly 24.0.1-Final

The properties file (jboss-ejb-client.properties) used to establish the connection is :

jboss.naming.client.ejb.context=true

endpoint.name=client-endpoint

remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=true

remote.connections=default
remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.default.protocol=https-remoting
remote.connection.default.host=10.15.20.10
remote.connection.default.port=4480
remote.connection.default.username=ejbca
remote.connection.default.password=ejbca

The client source code is like here under :

System.setProperty( "jndi.properties", "C:\Users\fmosse\Documents\Development\ejb-client\src\main\java\jboss-ejb-client.properties" );
System.setProperty( "jboss.ejb.client.properties.file.path",
"C:\Users\fmosse\Documents\Development\ejb-client\src\main\java\jboss-ejb-client.properties" );
System.setProperty( "javax.net.ssl.trustStore", "C:\Users\fmosse\Documents\Development\ejb-client\src\main\java\truststore.jks" );

System.setProperty( "java.naming.factory.initial", "org.wildfly.naming.client.WildFlyInitialContextFactory" );
System.setProperty( "java.naming.factory.url.pkgs", "org.jboss.ejb.client.naming" );

InitialContext context = null;
try {
context = new InitialContext();
} catch( NamingException e ) {
System.out.println( "Unable to instanciate the context due to exeception : " + e);
throw new RuntimeException( e );
}

System.out.println( "Initial context done");

String ejbUrl = "ejb:ejbca/cesecore-ejb//CertificateProfileSessionBean!org.cesecore.certificates.certificateprofile.CertificateProfileSessionRemote";

CertificateProfileSessionRemote certificateProfile = null;
try {
certificateProfile = (CertificateProfileSessionRemote)context.lookup( ejbUrl );
} catch( NamingException e ) {
System.out.println( "Unable to lookup remote ejb due to exception : " + e);
throw new RuntimeException( e );
}

System.out.println( "Ejb remote lookup done");

int certificate_profile_id;
String certificate_profile="TLS Server Profile";
try {
certificate_profile_id = certificateProfile.getCertificateProfileId( certificate_profile);
System.out.println("Example result: " + certificate_profile_id);
} catch (Exception e) {
System.err.println("Error accessing remote bean");
e.printStackTrace();
throw new RuntimeException( e );
}

try {
certificate_profile_id = certificateProfile.getCertificateProfileId( certificate_profile);
System.out.println("Example result: " + certificate_profile_id);
} catch (Exception e) {
System.err.println("Error accessing remote bean");
e.printStackTrace();
throw new RuntimeException( e );
}

I also use the EjbRemoteHelper to get the remote object reference but get the same issue :

    System.setProperty( "jndi.properties", "C:\\Users\\fmosse\\Documents\\Development\\ejb-client\\src\\main\\java\\jboss-ejb-client.properties" );
    System.setProperty( "jboss.ejb.client.properties.file.path",
"C:\\Users\\fmosse\\Documents\\Development\\ejb-client\\src\\main\\java\\jboss-ejb-client.properties" );
    System.setProperty( "javax.net.ssl.trustStore", "C:\\Users\\fmosse\\Documents\\Development\\ejb-client\\src\\main\\java\\truststore.jks" );

    System.setProperty("java.naming.factory.initial", "org.wildfly.naming.client.WildFlyInitialContextFactory");
    System.setProperty("java.naming.factory.url.pkgs", "org.jboss.ejb.client.naming");


    CertificateProfileSessionRemote certificateProfile= EjbRemoteHelper.INSTANCE.getRemoteSession( CertificateProfileSessionRemote.class );

    int certificate_profile_id;
    String certificate_profile="TLS Server Profile";
    try {
        certificate_profile_id = certificateProfile.getCertificateProfileId(certificate_profile);
        System.out.println("First call result: " + certificate_profile_id);
    } catch (Exception e) {
        System.err.println("Error accessing remote bean");
        e.printStackTrace();
        return;
    }

    try {
        certificateProfile= EjbRemoteHelper.INSTANCE.getRemoteSession( CertificateProfileSessionRemote.class );            certificate_profile_id = certificateProfile.getCertificateProfileId(certificate_profile);
        System.out.println("Second call result: " + certificate_profile_id);
    } catch (Exception e) {
        System.err.println("Error accessing remote bean");
        e.printStackTrace();
        return;
    }

=====================================================================================

Dependencies used for the compilation:
org.wildfly: wildfly-naming-client 2.0.0.Final
org.jboss: jboss-ejb-client 5.0.5.Final

and some package that have been built on our side for ejbca7 source code:
cesecore-common
cesecore-ejb-interface
ejbca-common

Best regards.

[primetomas]

No idea unfortunately. This is all WildFly and EJB remoting. Perhaps you can ask in the WildFly forum? It should be the same behavior if you create a trivial sample EJB application (like a hello world) and call that.

[fbredenbroecker]

I encountered the same problem (also the same stacktrace) accessing an EJB proxy of a different server application. As an addition in my case this happens only in a real client/server environment (client and server on different machines). If client and server are executed on the same system, you could use an EJB proxy multiple times regardless if remote+https is configured or not (standard remote+http).
Do you now have a solution to this problem?

Best regards

[primetomas]

The ejbca.sh CLI commands use remote EJB calls. If you can use the CLI it works as expected and you can copy code from there. If the CLI doesn't work, something was wrongly configured Remoting typically

try something easy like "bin/ejbca.sh ca listcas"

[fbredenbroecker]

Thank you for your direct reply, but in the meantime I solved my problem. It was based in a wildfly configuration problem
and does lead to a problem accessing an ejb proxy multiplte times (like described as a problem in this discussion).
Possibly my solution could also help the original issuer.

If wildfly is running in a cluster environment, the first proxy access will be handled by an EJBInvocationContext using
a communicaton context strong affinity = URI<protocol://address-or-hostname:port>, weak affinity = None which does lead
to a remote destination = URI<protocol://address-or-hostname:port> which is one of the configured remote connections from the
jboss-ejb-client.properties file.

The second (or further) call does use another mechanism. After the first call the EJB proxy of the EJBInvocationContext
will be relocated to strong affinity = Cluster "ejb", weak affinity = None to be able to switch to different
nodes of a cluster automatically

To resolve a corresponding address in this case the client does use the list of available cluster nodes which
has been read from the server during the initial communication establishment. This address is
created on each server-side using the configured socket binding section of the corresponding https connection.
In my case (wildfly 21) this is:

    <interfaces>
        <interface name="public">
            <inet-address value="${jboss.bind.address:127.0.0.1}"/>
        </interface>
    </interfaces>
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="https" port="${jboss.https.port:8443}">
            <client-mapping destination-address="${this-node-address-or-name:localhost}" destination-port="${jboss.https.port:8443}"/>
        </socket-binding>
    </socket-binding-group>

This configuration will pass back the this-node-address-or-name:8443 back to the client for the EJB dicovery
in the cluster mode.

I forgot to configure the client-mapping part in the socket-binding. In this case the server will pass back the address
from the corresponding interface section (public), which is 0.0.0.0 as the listening address in my configuration. But the client could not
establish a connection to remote+https://0.0.0.0:8443 which will lead to the Exception of the original issuer
at the second proxy access.

After adding this missing attributes, I have no more problems accessing an EJB proxy multiple times using remote+https.

Thanks for your reply and sorry for the inconvenience.

Best regards