Pushy on Wildfly works until suddenly it doesn't process messages anymore until entire wildfly container process is restarted

[arberg]

We've started using pushy. I must say, a nice and fast library. We are running it on Wildfly 22, using pushy com.eatthepath:pushy:0.15.4.

We see logs like these when it works:

Wrote payload on stream 45: <json message pushed>

and our ApnsClientMetricsListener.handleNotificationSent() gets invoked.

When the problem occurs these messages are no longer seen in the log and handleNotificationSent() isn't invoked, and but I can still see from other logs that we post message to pushy. We also never get callbacks in .whenCompleted() when the problem is there. When I investigate a heap-dump I can see 100-thousand of SimpleApnsPushNotification-messages in the ApnsChannelPool (or rather an anonymous subclass), indicating that it stopped processing them.

In wildfly it should be enough to restart a module to get it working again, unless either all the threads are busy (forever), or something is not being restarting when wildfly service is restarted. When it hung, I tried changing from using a wildfly ManagedExecutorService to the default pushy and that didn't make start processing messages unless I restarted the entire container.. This indicates to me pushy/netty is doing some deep stuff that's probably not legal wildfly. I have not disabled the unsafe api's yet, via JVM arguments.

Note that nothing else in our container is using this particular ManagedExecutorService which we have given to ApnsClient.

I noticed here that other's are using pushy on wildfly with apparent success, according to other old closed issues.

When we saw the problem in prod we didn't close the connection, and created unreasonably many apns'clients. I have seen it once again in test where we only create one ApnsClient in a @Singleton bean and closes it on shutdown, so I don't think that was the problem.

Any thoughts?

I have attached the full java file (renamed to txt), here's an abbreviated version with the most essential parts:

@Singleton
public class CloudPushAppleService {
    @Resource
    private ManagedExecutorService executorService;

    private ApnsClient prodApnsClient;
    private ApnsClient devApnsClient;

    @PostConstruct
    private void onStartup() {
        prodApnsClient = setupApnsClient(false);
        devApnsClient = setupApnsClient(true);
    }

    @PreDestroy
    private void onShutdown() {
        prodApnsClient.close();
        devApnsClient.close();
    }

    private ApnsClient setupApnsClient(boolean sandbox) {
        NioEventLoopGroup eventLoopGroup = new NioEventLoopGroup(1, executorService);
        return new ApnsClientBuilder()
                .setApnsServer(sandbox ? ApnsClientBuilder.DEVELOPMENT_APNS_HOST : ApnsClientBuilder.PRODUCTION_APNS_HOST)
                .setSigningKey(ApnsSigningKey.loadFromInputStream(new ByteArrayInputStream(key.getBytes()), teamId, keyId))
                .setEventLoopGroup(eventLoopGroup)
                .setMetricsListener(new BnrAppleMetrics(sandbox))
                .build();
    }

    public void sendLiveActivityPushNotification(PushNotificationMessage message) throws ServiceException {
        final ApnsPayloadBuilder payloadBuilder = new SimpleApnsPayloadBuilder();

        ...

        LOGGER.trace("Sending apple live activity push notification\n{}", appIsProduction ? "production" : "dev");

        final PushNotificationFuture<SimpleApnsPushNotification, PushNotificationResponse<SimpleApnsPushNotification>>
                sendNotificationFuture = apnsClient.sendNotification(pushNotification);

        final Map<String, String> logThreadContextMap = LogThreadContext.getMap();
        sendNotificationFuture.whenComplete((response, cause) -> {
            if (response != null) {
                if (response.isAccepted()) {
                    LOGGER.trace("Live activity push notification completed");
                } else {
                    .... log error
                }                
            } else {
                ... log error
            }
        });
    }

    private static class BnrAppleMetrics implements ApnsClientMetricsListener {
        private final String devProd;

        public BnrAppleMetrics(boolean sandbox) {
            this.devProd = sandbox ? "dev-sandbox" : "prod";
        }

        @Override
        public void handleWriteFailure(ApnsClient apnsClient, long count) {
            LogThreadContext.putEndPoint(END_POINT_NAME);
            LOGGER.trace("{} handleWriteFailure: {}", devProd, count);
        }

        @Override
        public void handleNotificationSent(ApnsClient apnsClient, long count) {
            LogThreadContext.putEndPoint(END_POINT_NAME);
            LOGGER.trace("{} handleNotificationSent: {}", devProd, count);
        }
        @Override
        public void handleNotificationAccepted(ApnsClient apnsClient, long count) {
            LogThreadContext.putEndPoint(END_POINT_NAME);
            LOGGER.trace("{} handleNotificationAccepted: {}", devProd, count);
        }
        @Override
        public void handleNotificationRejected(ApnsClient apnsClient, long count) {
            LogThreadContext.putEndPoint(END_POINT_NAME);
            LOGGER.trace("{} handleNotificationRejected: {}", devProd, count);
        }
        @Override
        public void handleConnectionAdded(ApnsClient apnsClient) {
            LogThreadContext.putEndPoint(END_POINT_NAME);
            LOGGER.trace("{} handleConnectionAdded", devProd);
        }
        @Override
        public void handleConnectionRemoved(ApnsClient apnsClient) {
            LogThreadContext.putEndPoint(END_POINT_NAME);
            LOGGER.trace("{} handleConnectionRemoved", devProd);
        }
        @Override
        public void handleConnectionCreationFailed(ApnsClient apnsClient) {
            LogThreadContext.putEndPoint(END_POINT_NAME);
            LOGGER.trace("{} handleConnectionCreationFailed", devProd);
        }
    }
}

Related but unimportant, I enabled the logging like this in log4j:

        <logger name="com.eatthepath.pushy" level="TRACE">
            <AppenderRef ref="ApplePushTraceFile"/>
        </logger>
        <logger name="io.netty" level="TRACE">
            <AppenderRef ref="ApplePushTraceFile"/>
        </logger>

CloudPushAppleService.java.txt