Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug Report: CDI deployment failure for 6.2025.2 #7201

Open
RInverid opened this issue Feb 19, 2025 · 33 comments · May be fixed by #7212
Open

Bug Report: CDI deployment failure for 6.2025.2 #7201

RInverid opened this issue Feb 19, 2025 · 33 comments · May be fixed by #7212
Assignees
Labels
Status: Open Issue has been triaged by the front-line engineers and is being worked on verification Type: Bug Label issue as a bug defect

Comments

@RInverid
Copy link

Brief Summary

deployment failure because some internal string is unexpectedly null:

org.glassfish.deployment.common.DeploymentException: CDI deployment failure:Cannot invoke "String.contains(java.lang.CharSequence)" because "beanArchiveId" is null -- Cannot invoke "String.contains(java.lang.CharSequence)" because "beanArchiveId" is null at org.jboss.weld.bootstrap.BeanDeployments.getFinalId(BeanDeployments.java:39) at org.jboss.weld.bootstrap.WeldStartup.startContainer(WeldStartup.java:189) at org.jboss.weld.bootstrap.WeldBootstrap.startContainer(WeldBootstrap.java:73) at org.glassfish.weld.WeldDeployer.startWeldBootstrap(WeldDeployer.java:578) at org.glassfish.weld.WeldDeployer.processApplicationLoaded(WeldDeployer.java:533) at org.glassfish.weld.WeldDeployer.event(WeldDeployer.java:422) at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:135) at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:344) at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:580) at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:570) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552) at java.base/java.security.AccessController.doPrivileged(AccessController.java:399) at java.base/javax.security.auth.Subject.doAs(Subject.java:376) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551) at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582) at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574) at java.base/java.security.AccessController.doPrivileged(AccessController.java:399) at java.base/javax.security.auth.Subject.doAs(Subject.java:376) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755) at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.executeCommand(CommandExecutorImpl.java:183) at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.run(CommandExecutorImpl.java:96) at fish.payara.boot.runtime.BootCommand.execute(BootCommand.java:69) at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:166) at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:160) at com.sun.enterprise.v3.bootstrap.BootCommandService.doBootCommands(BootCommandService.java:81) at com.sun.enterprise.v3.bootstrap.BootCommandService.postConstruct(BootCommandService.java:91) at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303) at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351) at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:466) at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281) at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65) at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2103) at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93) at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)

Looks like this might've been introduced by #7032 @lprimak

Expected Outcome

My app deploying normally, as it would on 6.2025.1

Current Outcome

Deployment failure

Reproducer

TBD, will work on creating one

Operating System

Debian Linux 12

JDK Version

OpenJDK 17

Payara Distribution

Payara Server Full Profile

@RInverid RInverid added Status: Open Issue has been triaged by the front-line engineers and is being worked on verification Type: Bug Label issue as a bug defect labels Feb 19, 2025
@lprimak
Copy link
Contributor

lprimak commented Feb 19, 2025

Do you have a reproducer?
Waiting for a reproducer...

@RInverid
Copy link
Author

RInverid commented Feb 20, 2025

No I am not able to build a small reproducer, I tried recreating the structure of my project but I have no idea what the issue could be, so not really able to produce any results

@lprimak
Copy link
Contributor

lprimak commented Feb 20, 2025

I can try running your app if that's at all possible. Is it using containers at all?

@lprimak
Copy link
Contributor

lprimak commented Feb 20, 2025

Is your app a WAR or EAR? Can you post a "file tree" from the WAR file?

@RInverid
Copy link
Author

Hi, sorry for forgetting to respond. Yeah, sharing the ear isn't an option sadly, but I can share the file tree for the ear. I'll be anonymising some stuff, is there anything specifically you're looking for?

@lprimak
Copy link
Contributor

lprimak commented Feb 22, 2025

Can you try the fix in #7211 please?
I am just guessing here since there is no reproducer, but hoping for the best

You can try either building the branch / PR or get the pre-built here:
https://nexus.hope.nyc.ny.us/repository/maven-releases/fish/payara/distributions/payara/6.2025.2-1/payara-6.2025.2-1.zip

@RInverid
Copy link
Author

sadly looks like the same issue is still present:

[#|2025-02-24T10:08:41.618+0000|SEVERE|Payara 6.2025.3|javax.enterprise.system.core|_ThreadID=44;_ThreadName=RunLevelControllerThread-1740391673967;_TimeMillis=1740391721618;_LevelValue=1000;_MessageID=NCLS-CORE-00026;| Exception during lifecycle processing org.glassfish.deployment.common.DeploymentException: CDI deployment failure:Cannot invoke "String.contains(java.lang.CharSequence)" because "beanArchiveId" is null -- Cannot invoke "String.contains(java.lang.CharSequence)" because "beanArchiveId" is null at org.jboss.weld.bootstrap.BeanDeployments.getFinalId(BeanDeployments.java:39) at org.jboss.weld.bootstrap.WeldStartup.startContainer(WeldStartup.java:189) at org.jboss.weld.bootstrap.WeldBootstrap.startContainer(WeldBootstrap.java:73) at org.glassfish.weld.WeldDeployer.startWeldBootstrap(WeldDeployer.java:577) at org.glassfish.weld.WeldDeployer.processApplicationLoaded(WeldDeployer.java:532) at org.glassfish.weld.WeldDeployer.event(WeldDeployer.java:421) at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:135) at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:344) at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:580) at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:570) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552) at java.base/java.security.AccessController.doPrivileged(AccessController.java:399) at java.base/javax.security.auth.Subject.doAs(Subject.java:376) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551) at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582) at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574) at java.base/java.security.AccessController.doPrivileged(AccessController.java:399) at java.base/javax.security.auth.Subject.doAs(Subject.java:376) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755) at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.executeCommand(CommandExecutorImpl.java:183) at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.run(CommandExecutorImpl.java:96) at fish.payara.boot.runtime.BootCommand.execute(BootCommand.java:69) at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:166) at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:160) at com.sun.enterprise.v3.bootstrap.BootCommandService.doBootCommands(BootCommandService.java:81) at com.sun.enterprise.v3.bootstrap.BootCommandService.postConstruct(BootCommandService.java:91) at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303) at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351) at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:466) at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281) at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65) at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2103) at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93) at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) |#]

I tried attaching a debugger, and saw that the DeploymentImpl constructor has a return on line 168, if libJarRootBDAs is not empty. This is the case for our project, which causes the deploymentImpl constructor to not run any code that sets the contextId. Does this info help? do you need to me to look at anything else?

@TheOnlyAl
Copy link

Hi.

I have the exact same problem with one of our deployments. My analysis of the problem matches with that of @RInverid
I also could not build a reproducer for now, but maybe will try again tomorrow.

@mkarg
Copy link
Contributor

mkarg commented Feb 24, 2025

We also have massive problems deploying our EAR (containing lots of WARs) on 6.2025.2, while 6.2025.1 was pretty fine.

Edit: We also have the problem that if we succeed to deploy on 6.2025.2 (after many attempts and very long waiting, it works "eventually"), then at some point we fail to @Inject a simple POJO suddenly... (always on the same target, which is in no way different to other targets). It seems your CDI change somehow makes that WELD blind for our producer, but just for this specific injection target...?! Weird... 🤔

WELD-001408: Unsatisfied dependencies for type DeploymentPlan with qualifiers @default

@lprimak
Copy link
Contributor

lprimak commented Feb 25, 2025

@TheOnlyAl
Copy link

Hi. For my deployment this fix sadly does not work. I am getting the same deployment error as for #7214:

[2025-02-25T07:43:57.630+0100] [Payara 6.2025.3] [SEVERE] [NCLS-CORE-00026] [javax.enterprise.system.core] [tid: _ThreadID=174 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1740465837630] [levelValue: 1000] [[
  Exception during lifecycle processing
org.glassfish.deployment.common.DeploymentException: CDI deployment failure:null -- null
        at com.sun.enterprise.deployment.Application.getPostConstructDescriptors(Application.java:624)
        at com.sun.enterprise.deployment.BundleDescriptor.getPostConstructDescriptorByClass(BundleDescriptor.java:491)
        at com.sun.enterprise.deployment.BundleDescriptor.getInjectionInfoByClass(BundleDescriptor.java:479)
        at com.sun.enterprise.deployment.Application.getInjectionInfoByClass(Application.java:661)
        at org.glassfish.weld.services.InjectionServicesImpl.registerInjectionTarget(InjectionServicesImpl.java:248)
        at org.jboss.weld.manager.InjectionTargetFactoryImpl.postProcess(InjectionTargetFactoryImpl.java:159)
        at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:94)
        at org.jboss.weld.bean.ManagedBean.<init>(ManagedBean.java:102)
        at org.jboss.weld.bean.ManagedBean.of(ManagedBean.java:82)
        at org.jboss.weld.bootstrap.AbstractBeanDeployer.createManagedBean(AbstractBeanDeployer.java:275)
        at org.jboss.weld.bootstrap.BeanDeployer.createClassBean(BeanDeployer.java:278)
        at org.jboss.weld.bootstrap.BeanDeployer.createClassBeans(BeanDeployer.java:256)
        at org.jboss.weld.bootstrap.BeanDeployment.createBeans(BeanDeployment.java:256)
        at org.jboss.weld.bootstrap.WeldStartup.deployBeans(WeldStartup.java:435)
        at org.jboss.weld.bootstrap.WeldBootstrap.deployBeans(WeldBootstrap.java:87)
        at org.glassfish.weld.WeldDeployer.startWeldBootstrap(WeldDeployer.java:586)
        at org.glassfish.weld.WeldDeployer.processApplicationLoaded(WeldDeployer.java:532)
        at org.glassfish.weld.WeldDeployer.event(WeldDeployer.java:421)
        at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:135)
        at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:344)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:573)
        at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:570)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:376)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:376)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755)
        at org.glassfish.admin.rest.resources.admin.CommandResource.executeCommand(CommandResource.java:409)
        at org.glassfish.admin.rest.resources.admin.CommandResource.execCommandMultInMultOut(CommandResource.java:258)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367)
        at org.glassfish.admin.rest.adapter.JerseyContainerCommandService$3.service(JerseyContainerCommandService.java:179)
        at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:189)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
        at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174)
        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153)
        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
        at org.glassfish.grizzly.portunif.PUFilter.handleRead(PUFilter.java:193)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
        at org.glassfish.grizzly.portunif.PUFilter.handleRead(PUFilter.java:193)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515)
        at java.base/java.lang.Thread.run(Thread.java:840)
]]

[2025-02-25T07:43:57.634+0100] [Payara 6.2025.3] [SEVERE] [] [javax.enterprise.system.core] [tid: _ThreadID=174 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1740465837634] [levelValue: 1000] [[
  Exception while loading the app]]

[2025-02-25T07:43:57.672+0100] [Payara 6.2025.3] [SEVERE] [] [javax.enterprise.system.core] [tid: _ThreadID=174 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1740465837672] [levelValue: 1000] [[
  Exception while loading the app : CDI deployment failure:null -- null]]

@RInverid
Copy link
Author

RInverid commented Feb 25, 2025

@lprimak I built it myself by checking out flowlogix:fix-cdi-context-npe and merging in flowlogix:fix-deployment-bugs. Sadly, it did not work - with a different error:

[#|2025-02-25T09:27:22.102+0000|SEVERE|Payara 6.2025.3|javax.enterprise.system.core|_ThreadID=45;_ThreadName=RunLevelControllerThread-1740475584638;_TimeMillis=1740475642102;_LevelValue=1000;_MessageID=NCLS-CORE-00026;| Exception during lifecycle processing org.glassfish.deployment.common.DeploymentException: CDI deployment failure:null -- null at com.sun.enterprise.deployment.Application.getPostConstructDescriptors(Application.java:624) at com.sun.enterprise.deployment.BundleDescriptor.getPostConstructDescriptorByClass(BundleDescriptor.java:491) at com.sun.enterprise.deployment.BundleDescriptor.getInjectionInfoByClass(BundleDescriptor.java:479) at com.sun.enterprise.deployment.Application.getInjectionInfoByClass(Application.java:661) at org.glassfish.weld.services.InjectionServicesImpl.registerInjectionTarget(InjectionServicesImpl.java:248) at org.jboss.weld.manager.InjectionTargetFactoryImpl.postProcess(InjectionTargetFactoryImpl.java:159) at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:94) at org.jboss.weld.bean.ManagedBean.<init>(ManagedBean.java:102) at org.jboss.weld.bean.ManagedBean.of(ManagedBean.java:82) at org.jboss.weld.bootstrap.AbstractBeanDeployer.createManagedBean(AbstractBeanDeployer.java:275) at org.jboss.weld.bootstrap.BeanDeployer.createClassBean(BeanDeployer.java:278) at org.jboss.weld.bootstrap.BeanDeployer.createClassBeans(BeanDeployer.java:256) at org.jboss.weld.bootstrap.BeanDeployment.createBeans(BeanDeployment.java:256) at org.jboss.weld.bootstrap.WeldStartup.deployBeans(WeldStartup.java:435) at org.jboss.weld.bootstrap.WeldBootstrap.deployBeans(WeldBootstrap.java:87) at org.glassfish.weld.WeldDeployer.startWeldBootstrap(WeldDeployer.java:586) at org.glassfish.weld.WeldDeployer.processApplicationLoaded(WeldDeployer.java:532) at org.glassfish.weld.WeldDeployer.event(WeldDeployer.java:421) at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:135) at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:344) at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:573) at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:570) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552) at java.base/java.security.AccessController.doPrivileged(AccessController.java:399) at java.base/javax.security.auth.Subject.doAs(Subject.java:376) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551) at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582) at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574) at java.base/java.security.AccessController.doPrivileged(AccessController.java:399) at java.base/javax.security.auth.Subject.doAs(Subject.java:376) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755) at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.executeCommand(CommandExecutorImpl.java:183) at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.run(CommandExecutorImpl.java:96) at fish.payara.boot.runtime.BootCommand.execute(BootCommand.java:69) at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:166) at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:160) at com.sun.enterprise.v3.bootstrap.BootCommandService.doBootCommands(BootCommandService.java:81) at com.sun.enterprise.v3.bootstrap.BootCommandService.postConstruct(BootCommandService.java:91) at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303) at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351) at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:466) at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281) at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65) at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2103) at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93) at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) |#]

Again, sorry for not being able to share a reproducer, but if I can help in any way by debugging & gathering more info, or sharing the file structure of our ear, I would be more than glad to help.

@lprimak
Copy link
Contributor

lprimak commented Feb 25, 2025

Please share file structure. Thank you

@TheOnlyAl
Copy link

I just checked. You can also see the exact same problem when you try to deploy the reproducer from #7214 Hope this helps.

@lprimak
Copy link
Contributor

lprimak commented Feb 25, 2025

@RInverid
Copy link
Author

I have to build the Payara build myself, I just took #7211 and #7212 and merged them just like last time. Sadly I got the same stack trace:

[#|2025-02-26T08:48:02.631+0000|SEVERE|Payara 6.2025.3|javax.enterprise.system.core|_ThreadID=45;_ThreadName=RunLevelControllerThread-1740559625784;_TimeMillis=1740559682631;_LevelValue=1000;_MessageID=NCLS-CORE-00026;| Exception during lifecycle processing org.glassfish.deployment.common.DeploymentException: CDI deployment failure:null -- null at com.sun.enterprise.deployment.Application.getPostConstructDescriptors(Application.java:624) at com.sun.enterprise.deployment.BundleDescriptor.getPostConstructDescriptorByClass(BundleDescriptor.java:491) at com.sun.enterprise.deployment.BundleDescriptor.getInjectionInfoByClass(BundleDescriptor.java:479) at com.sun.enterprise.deployment.Application.getInjectionInfoByClass(Application.java:661) at org.glassfish.weld.services.InjectionServicesImpl.registerInjectionTarget(InjectionServicesImpl.java:248) at org.jboss.weld.manager.InjectionTargetFactoryImpl.postProcess(InjectionTargetFactoryImpl.java:159) at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:94) at org.jboss.weld.bean.ManagedBean.<init>(ManagedBean.java:102) at org.jboss.weld.bean.InterceptorImpl.<init>(InterceptorImpl.java:64) at org.jboss.weld.bean.InterceptorImpl.of(InterceptorImpl.java:60) at org.jboss.weld.bootstrap.AbstractBeanDeployer.createInterceptor(AbstractBeanDeployer.java:288) at org.jboss.weld.bootstrap.BeanDeployer.createClassBean(BeanDeployer.java:275) at org.jboss.weld.bootstrap.BeanDeployer.createClassBeans(BeanDeployer.java:256) at org.jboss.weld.bootstrap.BeanDeployment.createBeans(BeanDeployment.java:256) at org.jboss.weld.bootstrap.WeldStartup.deployBeans(WeldStartup.java:435) at org.jboss.weld.bootstrap.WeldBootstrap.deployBeans(WeldBootstrap.java:87) at org.glassfish.weld.WeldDeployer.startWeldBootstrap(WeldDeployer.java:607) at org.glassfish.weld.WeldDeployer.processApplicationLoaded(WeldDeployer.java:538) at org.glassfish.weld.WeldDeployer.event(WeldDeployer.java:425) at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:135) at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:344) at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:573) at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:570) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552) at java.base/java.security.AccessController.doPrivileged(AccessController.java:399) at java.base/javax.security.auth.Subject.doAs(Subject.java:376) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551) at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582) at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574) at java.base/java.security.AccessController.doPrivileged(AccessController.java:399) at java.base/javax.security.auth.Subject.doAs(Subject.java:376) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755) at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.executeCommand(CommandExecutorImpl.java:183) at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.run(CommandExecutorImpl.java:96) at fish.payara.boot.runtime.BootCommand.execute(BootCommand.java:69) at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:166) at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:160) at com.sun.enterprise.v3.bootstrap.BootCommandService.doBootCommands(BootCommandService.java:81) at com.sun.enterprise.v3.bootstrap.BootCommandService.postConstruct(BootCommandService.java:91) at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303) at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351) at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:466) at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281) at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65) at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2103) at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93) at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144) at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) |#]

For the ear file structure, a shortened summary:
file.ear

  • 35 jars
  • 14 wars
  • lib folder
    • A lot of jars
  • META-INF folder
    • application.xml
    • glassfish-application.xml
    • MANIFEST.MF

@lprimak
Copy link
Contributor

lprimak commented Feb 26, 2025

@RInverid Thank you for the update. Will keep working on it.

Can I get a more detailed summary?

Perhaps output of tree -a command?
With all the WARs un-packed as well?

Thank you

@lprimak
Copy link
Contributor

lprimak commented Feb 26, 2025

am sorry for the bit of a short-term pain, but in the long-term, this is leading to #7165 performance,
which should benefit everyone. I am hoping for at least 4X deploy performance improvements.

@lprimak
Copy link
Contributor

lprimak commented Feb 26, 2025

Good news is that with @TheOnlyAl 's reproducer, I can see your initial error, and that the changes do fix it. I will close the initial PR and incorporate all the fixes into a single PR. Still work in progress, but progress is finally being made I think... he he

@lprimak
Copy link
Contributor

lprimak commented Feb 26, 2025

@RInverid
Copy link
Author

RInverid commented Feb 27, 2025

I just built #7212, I assume thats the branch you mean. Failed again in new unique ways, 4 different stacktraces that seem relevant, so I collected them in a gist: https://gist.github.com/RInverid/1a923ed6764cf827090ae97305a2530d

Sorry about the more detailed overview, can't do that. I can show an example unpacked war:

example.war.d
├── META-INF
│   ├── MANIFEST.MF
│   └── maven
│       └── package
│           └── folder
│               ├── pom.properties
│               └── pom.xml
└── WEB-INF
    ├── beans.xml
    ├── classes
    │   └── package
    │                           ├── monitor
    │                           │   ├── Metrics.class
    │                           │   └── MetricsFactory.class
    │                           └── resources
    │                               ├── RootApplication.class
    │                               └── Streams.class
    ├── glassfish-web.xml
    ├── lib
    │   ├── dep.jar
    │   ├── dep.jar
    │   └── jakarta.json-1.1.5.payara-p1.jar
    └── web.xml

@lprimak
Copy link
Contributor

lprimak commented Feb 27, 2025

I have another fix pending. Getting close I think

@lprimak
Copy link
Contributor

lprimak commented Feb 27, 2025

@RInverid
Copy link
Author

It works! :o fantastic, thanks for your hard work

@lprimak
Copy link
Contributor

lprimak commented Feb 28, 2025

#Awesome! Now I can finally get back to performance PR :). Thank you for your help.

@mkarg
Copy link
Contributor

mkarg commented Mar 3, 2025

We also have the problem that if we succeed to deploy on 6.2025.2 (after many attempts and very long waiting, it works "eventually"), then at some point we fail to @Inject a simple POJO suddenly... (always on the same target, which is in no way different to other targets). It seems your CDI change somehow makes that WELD blind for our producer, but just for this specific injection target...?! Weird... 🤔

WELD-001408: Unsatisfied dependencies for type DeploymentPlan with qualifiers @default

The exact same EAR worked fine on 6.2025.1, but it still fails on 6.2025.2-6 (!). So it seems, this still is related to the 6.2025.2-CDI-changes, but unrelated form @lprimak's already fixed bugs. Shall we enable some particular logger or run a particular test drive? 🤔

org.jboss.weld.exceptions.IllegalArgumentException: WELD-001408: Unsatisfied dependencies for type DeploymentPlan with qualifiers @Default
  at injection point [BackedAnnotatedField] @Inject private de.quipsy.inspections.server.api.references.InspectionPointReference$Adapter.deploymentPlan
  at de.quipsy.inspections.server.api.references.InspectionPointReference$Adapter.deploymentPlan(Adapter.java:0)

        at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:83)
        at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:70)
        at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:51)
        at org.eclipse.yasson.internal.components.BeanManagerInstanceCreator.lambda$getOrCreateComponent$0(BeanManagerInstanceCreator.java:69)

@lprimak
Copy link
Contributor

lprimak commented Mar 3, 2025

I think reproducer is key here. Also I think there is a special meaning in Payara with "DeploymentPlan". Can you rename the bean to see if that helps?

@lprimak
Copy link
Contributor

lprimak commented Mar 3, 2025

@mkarg you should open a separate bug for this, since there is no reproducer (yet, I hope)

@lprimak
Copy link
Contributor

lprimak commented Mar 3, 2025

@mkarg
I have some ideas / guesses... First of all, none of your stack traces include any Payara code. Looks like Weld, yasson and quipsy code only. This is definitely not enough information to figure out what's going on.

So here is my idea...
New code separates Weld deployments between different WARs-in-EARs.
One WAR cannot see another WAR's objects inside EAR, fully separating them.
This would cause issues when multiple WARs in an EAR used the same CDI dependencies in them.
It is the correct behavior per the SPECs.

Old code did not do this, and "mashed" all the CDI beans together in the same EAR, even if they are present in separate WARs. Due to this, "old code" could see beans from one WAR to another.

My guess is that your code is using this "bug" / old behavior to rely on seeing all the beans in the EAR, even the ones you are not supposed to.

@mkarg
Copy link
Contributor

mkarg commented Mar 3, 2025

Renaming of both, injected POJO and injection point, did not change the behavior.

Yes, the EAR actually contains many WARs, but thanks to the renaming the problem also happens when injected POJO, its CDI producer, and the CDI injection target live in the very same WAR -- and only in that one. Still the problem occurs though. It might be related to that idea anyways: Inside the WAR, no classes exists, but all classes live in separate JARs inside the lib folder of that very WAR. Maybe you changed something in that area, too, which now separates the producer libs from the injection target libs?

Anyways, will set up a reproducer tomorrow and open a separate ticket for this.

@lprimak
Copy link
Contributor

lprimak commented Mar 3, 2025

Thank you! Appreciate it.

@mkarg
Copy link
Contributor

mkarg commented Mar 4, 2025

Still working on the reproducer, and the situation is rather weird: As soon as I simply remove all other WARs from the EAR but keep solely the failing one, it starts working perfectly and the error is gone completely! This proofs that the bug indeed is related to the "mixing CDI among WARs" paradigm, but apparently in the reverse direction: The EAR wants separated namespaces, but 6.2025.2-6 mixes them?! 😲

@mkarg
Copy link
Contributor

mkarg commented Mar 4, 2025

Finished with the reproducer, please continue discussion in the new issue: #7233

@lprimak Hope this helps identifying and fixing the problem! 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Open Issue has been triaged by the front-line engineers and is being worked on verification Type: Bug Label issue as a bug defect
Projects
None yet
6 participants