Bug Report: asadmin undeploy timeout

[mkarg]

Brief Summary

asadmin undeploy MyApp runs into timeout on Full-CE 6.2025.2, but worked fine on Full-CE 6.2025.1

Expected Outcome

My-App is undeployed without timeout / error.

Current Outcome

No response from Domain Admin Server after 600 seconds.
The command is either taking too long to complete or the server has failed.
Please see the server log files for command status.
Command undeploy failed.

[2025-02-25T07:58:23.913+0100] [Payara 6.2025.2] [SCHWERWIEGEND] [] [javax.enterprise.system.util] [tid: _ThreadID=90 _ThreadName=admin-thread-pool::admin-listener(1)] [timeMillis: 1740466703913] [levelValue: 1000] [[
  Thread Interrupted Exception
java.lang.InterruptedException: sleep interrupted
        at java.base/java.lang.Thread.sleep0(Native Method)
        at java.base/java.lang.Thread.sleep(Unknown Source)
        at com.sun.enterprise.util.io.FileUtils.doWithRetry(FileUtils.java:780)
        at com.sun.enterprise.util.io.FileUtils.internalDeleteFile(FileUtils.java:627)
        at com.sun.enterprise.util.io.FileUtils.deleteFileWithWaitLoop(FileUtils.java:589)
        at com.sun.enterprise.deploy.shared.FileArchive.deleteDir(FileArchive.java:608)
        at com.sun.enterprise.deploy.shared.FileArchive.deleteDir(FileArchive.java:605)
        at com.sun.enterprise.deploy.shared.FileArchive.deleteDir(FileArchive.java:605)
        at com.sun.enterprise.deploy.shared.FileArchive.deleteDir(FileArchive.java:605)
        at com.sun.enterprise.deploy.shared.FileArchive.delete(FileArchive.java:237)
        at org.glassfish.deployment.admin.UndeployCommand.execute(UndeployCommand.java:467)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAs(Unknown Source)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAs(Unknown Source)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755)
        at org.glassfish.admin.rest.resources.admin.CommandResource.executeCommand(CommandResource.java:409)
        at org.glassfish.admin.rest.resources.admin.CommandResource.execCommandSimpInMultOut(CommandResource.java:236)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367)
        at org.glassfish.admin.rest.adapter.JerseyContainerCommandService$3.service(JerseyContainerCommandService.java:179)
        at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:189)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
        at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174)
        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153)
        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515)
        at java.base/java.lang.Thread.run(Unknown Source)
]]

• payara6\glassfish\domains\domain1\applications\My-App still contains all .war files from our .ear file, and all META.INF/lib/*.jar files in the unpacked .war folders. Tried to remove them using CLI commands, but OS says all of those is locked. asadmin list applications says My-App is undeployed. Shutdown of Payara service unlocks everything, could then use CLI commands to delete all of that. => This proofs that it is Payara is locking itself!

Reproducer

We will provide our EAR directly to an engineer if needed.

Operating System

Win 10 Pro

JDK Version

OpenJDK 64-Bit Server VM Zulu21.38+21-CA (build 21.0.5+11-LTS, mixed mode, sharing)

Payara Distribution

Payara Server Full Profile

[mkarg]

@lprimak As the very same EAR worked well from Payara 4 (certainly an older EAR) up to Payara 6.2025.1 I assume that your recent CDI changes has to do with this, as since 6.2025.2 the same EAR produces a runtime failure where a CDI producer is not called anymore. Any loggers you want me to enable and retry?

[lprimak]

Maybe... I can't see the candidate off the top of my head, but it's definitely possible.
The problem is that I don't have Windows and it's Windows-specific problems.
@Pandrex247 I remember you dealt with problems like this in the past.
Do you have a technique that I can borrow to figure out where these files get locked?

Thank you!

[lprimak]

@mkarg Does it happen only in EAR files? Or in WAR files as well?

[mkarg]

We do not have any standalone WARs. Do you think it is worth trying with an empty one?

[lprimak]

Yes, please try with empty WAR and empty EAR, just to isolate the problem a bit more.
Thank you!

[lprimak]

I was able to see the issue on mac using lsof
Not sure if it's related, but I will try to figure it out.

[lprimak]

Can you try the latest build please?

https://nexus.hope.nyc.ny.us/repository/maven-releases/fish/payara/distributions/payara/6.2025.2-6/payara-6.2025.2-6.zip

Thank you

[lprimak]

I was able to confirm that the attached PR actually closes all open files now and should fix this problem

[mkarg]

Sorry for the delay. Unfortunately I did not find the time to run a test with empty EAR / WAR, but apparently it is not needed anymore: Payara Full CE 6.2025.2-6 perfectly undeploys our EAR without any problem! ❤ Lenny, you are my hero! 🦸‍♂