Fileleak prevents application undeploy (caused by "WebappClassLoader.getJarFiles")

[WolfgangHG]

Sorry, this is a hard to reproduce issue, and I don't have an easy reproducer.

The https://github.com/arquillian/arquillian-extension-warp deploys as part of the test suite some war files to a WildFly/TomEE/Glassfish server. With Glassfish, undeploy fails to delete two jar files in "WEB-INF\lib" on windows, resulting in a delay of 20 seconds for each test until file deletion times out.
This happens with Glassfish 7.0.18

Using https://github.com/jenkinsci/lib-file-leak-detector/, I could track it down to two stacktraces that leave a file handle open. This tool prints at shutdown of Glassfish for each unclosed file the stack trace that opened it.

One is coming from JBoss Weld and I think I can send a pull request myself. Issue. https://issues.redhat.com/browse/WELD-2800

The other one is from Weld/Glassfish. Here is the output of "file-leak-detector" for one of the two jar files:

[2024-10-26T10:37:12.198647+02:00] [GF 7.0.18] [SEVERE] [] [jakarta.enterprise.logging.stderr] [tid: _ThreadID=93 _ThreadName=File handles dumping shutdown hook] [levelValue: 1000] [[
  #43 C:\Temp\glassfish7\glassfish\domains\domain1\applications\test\WEB-INF\lib\arquillian-testenricher-cdi-jakarta.jar by thread:admin-listener(2) on Sat Oct 26 10:36:32 CEST 2024
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
	at java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1328)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1294)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:742)
	at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:851)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:257)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:186)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:348)
	at org.glassfish.web.loader.JarFileManager.openJARs(JarFileManager.java:216)
	at org.glassfish.web.loader.JarFileManager.getJarFiles(JarFileManager.java:92)
	at org.glassfish.web.loader.WebappClassLoader.getJarFiles(WebappClassLoader.java:1043)
	at org.apache.naming.resources.WebDirContext.lookupFromJars(WebDirContext.java:287)
	at org.apache.naming.resources.WebDirContext.lookup(WebDirContext.java:133)
	at org.glassfish.web.loader.WebappClassLoader.findResources(WebappClassLoader.java:694)
	at com.sun.enterprise.loader.ResourceLocator.getResources(ResourceLocator.java:83)
	at org.glassfish.web.loader.WebappClassLoader.getResources(WebappClassLoader.java:843)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1196)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
	at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
	at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
	at java.xml/javax.xml.validation.SchemaFactoryFinder$2.run(SchemaFactoryFinder.java:339)
	at java.xml/javax.xml.validation.SchemaFactoryFinder$2.run(SchemaFactoryFinder.java:335)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:310)
	at java.xml/javax.xml.validation.SchemaFactoryFinder.findServiceProvider(SchemaFactoryFinder.java:335)
	at java.xml/javax.xml.validation.SchemaFactoryFinder._newFactory(SchemaFactoryFinder.java:217)
	at java.xml/javax.xml.validation.SchemaFactoryFinder.newFactory(SchemaFactoryFinder.java:144)
	at java.xml/javax.xml.validation.SchemaFactory.newInstance(SchemaFactory.java:245)
	at org.jboss.weld.xml.BeansXmlValidator.<init>(BeansXmlValidator.java:61)
	at org.jboss.weld.bootstrap.WeldBootstrap.<init>(WeldBootstrap.java:58)
	at org.glassfish.weld.WeldDeployer.load(WeldDeployer.java:219)
	at org.glassfish.weld.WeldDeployer.load(WeldDeployer.java:123)
	at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:198)
	at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:310)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:529)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:257)
	at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:471)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:574)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:570)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:389)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:569)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:600)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:592)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:389)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:591)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1484)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1862)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1738)
	at org.glassfish.admin.rest.utils.ResourceUtil.runCommand(ResourceUtil.java:233)
	at org.glassfish.admin.rest.utils.ResourceUtil.runCommand(ResourceUtil.java:213)
	at org.glassfish.admin.rest.utils.ResourceUtil.runCommand(ResourceUtil.java:253)
	at org.glassfish.admin.rest.resources.TemplateListOfResource.createResource(TemplateListOfResource.java:305)
	at org.glassfish.admin.rest.resources.TemplateListOfResource.post(TemplateListOfResource.java:142)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367)
	at org.glassfish.admin.rest.adapter.RestAdapter$2.service(RestAdapter.java:291)
	at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:159)
	at org.glassfish.admin.rest.adapter.RestManagementAdapter.service(RestManagementAdapter.java:42)
	at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:429)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:143)
	at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174)
	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153)
	at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)
	at org.glass]]

Do you have any idea about this?

Attached is the web app war file (rename it to "test.war") - but the issue does not happen if I just deploy/undeploy the app with the web console.
test.zip
The affected locked files are "arquillian-testenricher-cdi-jakarta.jar" and "arquillian-core.jar". Both define "META-INF\services".

More steps on reproducing it will follow in the next comment.

[WolfgangHG]

To reproduce (requires windows OS):

1. clone https://github.com/arquillian/arquillian-extension-warp. As we are in the process of migrating to JakartaEE10, there is no release available. Otherwise I might have created a standalone sample.
2. start a glassfish 7.0.18 server
3. run one single test in the arquillian-extension-warp testsuite:

   mvnw install -Pglassfish-remote -Dtest="org.jboss.arquillian.warp.ftest.cdi.CdiWarpTest" -Dsurefire.failIfNoSpecifiedTests=false
   

Now you should see that two files cannot be deleted.

To see the stacktrace where an unclosed file is opened

1. download "file-leak-detector-1.18-jar-with-dependencies.jar" from https://github.com/jenkinsci/lib-file-leak-detector/
2. launch glassfish
3. attach the leak detector:

   java -jar c:\path\to\file-leak-detector-1.18-jar-with-dependencies.jar 1234 strong,dumpatshutdown
   

   where "1234" is the id of the glassfish process
4. run the test suite, see step 3 above
5. shutdown glassfish. Now the leak detector prints the leaking stack traces.

[WolfgangHG]

Keep this issue on hold - I think it got it to work.
If you have no objections, it is more about upgrading Weld.

I applied a fix to Weld (weld/core#3069), built it and replaced "weld-osgi-bundle.jar" in my glassfish 7.0.18 installation with a renamed "weld-osgi-bundle-5.1.4-SNAPSHOT.jar".
After this, the same file leak was reported for arquillian (arquillian/arquillian-core#637). And after another fix, the file leak problem was gone.

So, hopefully it is really fixed now. Or do you see a problem caused by me replacing "weld-osgi-bundle.jar"?

[OndroMih]

What you did makes sense, @WolfgangHG. Thank you for those fixes! As soon as those fixes are released in Weld, we'll upgrade Weld in GlassFish.

[WolfgangHG]

Weld 5.1.4 was released last week. So, it would be great if the update could be integrated in a Glassfish 7.0.21 release.

[OndroMih]

Fixed by #25296

[WolfgangHG]

Many thanks!