Jetty ignores OutOfMemory

[nino-magazinovic]

Tested with jetty-9.4.10.v20180503.

OutOfMemoryError did not trigger application restart due to wrong jetty error handling:

2018-10-26 11:03:39.660:INFO:oejc.ResponseNotifier:HttpClient@1eb6195-88135: Exception while notifying listener com.esq.rmmagent.util.jettyhttpsender.JettyHttpSender$ContextExchangeEx@3b4738
java.lang.OutOfMemoryError: Java heap space

where org.eclipse.jetty.client.ResponseNotifier catches with Throwable and only logs error.

Same happens in:

2018-10-25 01:59:58.363:WARN:oejs.HttpChannel:qtp585932-39: /COM/do
java.lang.OutOfMemoryError: Java heap space

where org.eclipse.jetty.server.HttpChannel catches with Error and only logs error.

In both cases it infinitely logs error without crashing the server.

[sbordet]

If you get an OutOfMemoryError it's typically already too late - you either have a too small heap, or your application leaks memory.

I don't understand what do you expect that Jetty did instead.

Even if we leave that error unwind the stack, it will be likely on a Jetty thread, so the Jetty thread pool will log the error and nothing can be done by your application.
If you know how to handle the OutOfMemoryError, then you have to try/catch it in your application code.

[nino-magazinovic]

The OutOfMemoryError was caused by jetty websocket and httpclient #2655 and #2860 (accumulating beans).

Application is wrapped by tanuki service wrapper which can restart the application using wrapper.filter.trigger by monitoring the console output of the JVM.

Seems that it did not trigger because the log was dumped to the application log instead of console.

[sbordet]

Those Jetty issues have been fixed, so upgrade Jetty to the latest version.

Jetty logging can delegate to other logging frameworks that are very flexible in terms of where they write logs, so it's just a matter of configuring the logging framework you use properly.

[nino-magazinovic]

Thanks.

Jetty info logs are dumped to the console but a different file (defined in jetty-logging.xml) which was not scanned by the tanuki wrapper.
Issue resolved by targeting the same log file for console output (not using jetty-logging.xml).

[sbordet]

Closing as resolved.