Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Random crash in JavaThread "Monitor Deflation Thread" daemon #1035

Open
andresluuk opened this issue Mar 4, 2024 · 4 comments
Open

Random crash in JavaThread "Monitor Deflation Thread" daemon #1035

andresluuk opened this issue Mar 4, 2024 · 4 comments
Labels
bug Something isn't working jbs:needs-report Waiting for someone from our org to report to OpenJDK

Comments

@andresluuk
Copy link

Please provide a brief summary of the bug

We have seen the following crash a few times:

SIGSEGV (0xb) at pc=0x00007fb68c49bf18, pid=407379, tid=407743

JRE version: OpenJDK Runtime Environment Temurin-21.0.2+13 (21.0.2+13) (build 21.0.2+13-LTS)

Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.2+13 (21.0.2+13-LTS, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)

Problematic frame:

V [libjvm.so+0xcdef18] OopStorage::Block::release_entries(unsigned long, OopStorage*)+0x38

Did you test with the latest update version?

Yes

Please provide steps to reproduce where possible

Crash is random so can't make a repro. But I have other crash dumps from the same location saved locally.
A few caviatest:
For some reason we see the crash only on latest Wildflyes (29, 30, 31), not on other servers. (Jetty, Tomcat etc).
Crash happens in less then 1% of the server runs, but as we do many runs we see it every few days.
We have the jrebel java agent attached, so can't rule that out entirely, but usually crashes like that come from the JDK for us.

Expected Results

No crash.

Actual Results

JVM crash with a SIGSEGV

What Java Version are you using?

OpenJDK Runtime Environment Temurin-21.0.2+13 (21.0.2+13) (build 21.0.2+13-LTS)

What is your operating system and platform?

Ubuntu

How did you install Java?

We unzipped it from the zip installation.

Did it work before?

It has been crashing for a few months so we have seen it in the latest few JDK versions.

Did you test with other Java versions?

No response

Relevant log output

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fb68c49bf18, pid=407379, tid=407743
#
# JRE version: OpenJDK Runtime Environment Temurin-21.0.2+13 (21.0.2+13) (build 21.0.2+13-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.2+13 (21.0.2+13-LTS, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xcdef18]  OopStorage::Block::release_entries(unsigned long, OopStorage*)+0x38
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/zeroturnaround/jenkins-agent/workspace/javeleon-ejbtest-wildfly-latest/2f606c00/core.407379)
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
#

---------------  S U M M A R Y ------------

Command Line: -D[Standalone] -Djdk.serialFilter=maxbytes=10485760;maxdepth=128;maxarray=100000;maxrefs=300000 -DIS_CYGWIN=false -agentpath:/home/zeroturnaround/jenkins-agent/workspace/javeleon-ejbtest-wildfly-latest/2f606c00/tmp/jrebel/lib/libjrebel64.so -Drebel.usage_reporting=false -Drebel.log=true -Drebel.log.file=/home/zeroturnaround/jenkins-agent/workspace/javeleon-ejbtest-wildfly-latest/2f606c00/log/wildfly31preview_suiteQaEjb3_packaged-modules_rebel.log -Drebel.log.perf_on_exit=true -Drebel.log.dump_stack_traces=60 -Drebel.loggers.trace=OpenWebBeans -Drebel.enable_test_hacks=true -Drebel.log.licensing=false -Drebel.mixpanel.url=http://127.0.0.1:44443 -Drebel.fatal_licensing=true -Drebel.preferred.license=1 -Drebel.license.email=EC2zt-devel-ubuntui-08e61c34226742374@zeroturnaround.com -Drebel.temp.dir=/home/zeroturnaround/jenkins-agent/workspace/javeleon-ejbtest-wildfly-latest/2f606c00/tmp/wildfly31preview -Drebel.remoting.cacheDir=/home/zeroturnaround/jenkins-agent/workspace/javeleon-ejbtest-wildfly-latest/2f606c00/tmp/wildfly31preview/remoting -Drebel.ide.console_log=false -Xlog:gc -Drebel.license.url=https://licenses-staging.zeroturnaround.com/e002d327-1a90-4b47-8034-d8aa078c9ed4 -XX:ReservedCodeCacheSize=96m -XX:MaxMetaspaceSize=1536m -Xmx1324m -Xss256k -Drebel.jboss7_deploy_hack=true -Drebel.detect_closed_zip=true --add-exports=java.desktop/sun.awt=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldaps=ALL-UNNAMED --add-exports=jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.management/javax.management=ALL-UNNAMED --add-opens=java.naming/javax.naming=ALL-UNNAMED -Djava.security.manager=allow -Dorg.jboss.boot.log.file=/opt/containers/wildfly-preview-31.0.1.Final/standalone/log/server.log -Dlogging.configuration=file:/opt/containers/wildfly-preview-31.0.1.Final/standalone/configuration/logging.properties /opt/containers/wildfly-preview-31.0.1.Final/jboss-modules.jar -mp /opt/containers/wildfly-preview-31.0.1.Final/modules org.jboss.as.standalone -Djboss.home.dir=/opt/containers/wildfly-preview-31.0.1.Final -Djboss.server.base.dir=/opt/containers/wildfly-preview-31.0.1.Final/standalone --server-config=standalone-full.xml

Host: AMD EPYC 7R13 Processor, 8 cores, 15G, Ubuntu 20.04.6 LTS
Time: Sun Mar  3 19:22:56 2024 UTC elapsed time: 308.656561 seconds (0d 0h 5m 8s)

---------------  T H R E A D  ---------------

Current thread (0x000055e7d1ec50f0):  JavaThread "Monitor Deflation Thread" daemon [_thread_in_vm, id=407743, stack(0x00007fb6784eb000,0x00007fb67852c000) (260K)]

Stack: [0x00007fb6784eb000,0x00007fb67852c000],  sp=0x00007fb67852a910,  free space=254k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xcdef18]  OopStorage::Block::release_entries(unsigned long, OopStorage*)+0x38
V  [libjvm.so+0xcdf189]  OopStorage::release(oopDesc* const*)+0x89
V  [libjvm.so+0xcc8548]  ObjectMonitor::deflate_monitor()+0x248
V  [libjvm.so+0xf0e1a9]  ObjectSynchronizer::deflate_monitor_list(Thread*, LogStream*, elapsedTimer*, ObjectMonitorsHashtable*)+0x129
V  [libjvm.so+0xf0e364]  ObjectSynchronizer::deflate_idle_monitors(ObjectMonitorsHashtable*)+0x174
V  [libjvm.so+0xc941a7]  MonitorDeflationThread::monitor_deflation_thread_entry(JavaThread*, JavaThread*)+0x137
V  [libjvm.so+0x91a1d0]  JavaThread::thread_main_inner()+0x1e0
V  [libjvm.so+0xf5fd98]  Thread::call_run()+0xa8
V  [libjvm.so+0xcee34a]  thread_native_entry(Thread*)+0xda

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000238

Registers:
RAX=0x0000000000000001, RBX=0x000055e7d0aedb60, RCX=0x00000abcfc125002, RDX=0x000055e7d0aedb20
RSP=0x00007fb67852a910, RBP=0x00007fb67852a9f0, RSI=0x0000000000000004, RDI=0x0000000000000000
R8 =0x0000000000000000, R9 =0xfffffffffffffffe, R10=0x0000000000000000, R11=0x0000000000000000
R12=0x0000000000000004, R13=0x0000000000000000, R14=0x000055e7d0aedb20, R15=0x0000000000000238
RIP=0x00007fb68c49bf18, EFLAGS=0x0000000000010246, CSGSFS=0x002b000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e


Register to memory mapping:

RAX=0x0000000000000001 is an unknown value
RBX=0x000055e7d0aedb60 points into unknown readable memory: 0x0000000000000039 | 39 00 00 00 00 00 00 00
RCX=0x00000abcfc125002 is an unknown value
RDX=0x000055e7d0aedb20 points into unknown readable memory: 0x000055e7d0a61f10 | 10 1f a6 d0 e7 55 00 00
RSP=0x00007fb67852a910 is pointing into the stack for thread: 0x000055e7d1ec50f0
RBP=0x00007fb67852a9f0 is pointing into the stack for thread: 0x000055e7d1ec50f0
RSI=0x0000000000000004 is an unknown value
RDI=0x0 is null
R8 =0x0 is null
R9 =0xfffffffffffffffe is an unknown value
R10=0x0 is null
R11=0x0 is null
R12=0x0000000000000004 is an unknown value
R13=0x0 is null
R14=0x000055e7d0aedb20 points into unknown readable memory: 0x000055e7d0a61f10 | 10 1f a6 d0 e7 55 00 00
R15=0x0000000000000238 is an unknown value

Top of Stack: (sp=0x00007fb67852a910)
0x00007fb67852a910:   0000000000000000 00007fb68ccbcae0
0x00007fb67852a920:   0000000000000000 00000000000008be
0x00007fb67852a930:   00007fb68ccbcae0 0000000000000000
0x00007fb67852a940:   00007fb68cefa050 00007fb67852a960
0x00007fb67852a950:   00007fb67852a9a0 0000000000000000
0x00007fb67852a960:   00000000000008be 00007fb68ccbcad0
0x00007fb67852a970:   00007fb68ccbcaa8 000055e700000000
0x00007fb67852a980:   00007fb68ccbcaa0 2cb99b9875c1ca00
0x00007fb67852a990:   000055e7d1ec6340 000055e7d1ec50f0
0x00007fb67852a9a0:   00007fb67852aa10 00007fb68ccbcaa8
0x00007fb67852a9b0:   00007fb68ccbcad0 2cb99b9875c1ca00
0x00007fb67852a9c0:   0000000000000000 000055e7d0aedb60
0x00007fb67852a9d0:   0000000000000000 000055e7e0928010
0x00007fb67852a9e0:   000055e7e0928040 00007fb67852aba0
0x00007fb67852a9f0:   00007fb67852aa20 00007fb68c49c189
0x00007fb67852aa00:   000055e7d1ec50f0 000055e7dfdfb660
0x00007fb67852aa10:   0000000000000000 00007fb68cc613c8
0x00007fb67852aa20:   00007fb67852aa90 00007fb68c485548
0x00007fb67852aa30:   00007fb68c964a51 000055e7d1ec50f0
0x00007fb67852aa40:   00007fb67852aa90 00007fb68c6cad8c
0x00007fb67852aa50:   00007fb68cedfb80 00007fb68c964a42
0x00007fb67852aa60:   000055e7e030bf70 000055e7d1ec50f0
0x00007fb67852aa70:   00000000000002f2 000055e7e0928100
0x00007fb67852aa80:   00007fb68cc613c8 00007fb67852aba0
0x00007fb67852aa90:   00007fb67852aaf0 00007fb68c6cb1a9
0x00007fb67852aaa0:   00007fb68ccbc398 0000000000000000
0x00007fb67852aab0:   00007fb67852aba0 0000000000000000
0x00007fb67852aac0:   000055e7d1ec50f0 00007fb68ccbfc20
0x00007fb67852aad0:   00007fb68cc2c0e8 000055e7d1ec50f0
0x00007fb67852aae0:   0000000000000000 00007fb67852aba0
0x00007fb67852aaf0:   00007fb67852ad50 00007fb68c6cb364
0x00007fb67852ab00:   00007fb67852ac20 0000000000003a63 

Instructions: (pc=0x00007fb68c49bf18)
0x00007fb68c49be18:   0f 1f 84 00 00 00 00 00 83 68 78 01 eb c3 66 2e
0x00007fb68c49be28:   0f 1f 84 00 00 00 00 00 80 7f 24 00 75 0a c3 66
0x00007fb68c49be38:   0f 1f 84 00 00 00 00 00 55 48 89 e5 41 55 41 54
0x00007fb68c49be48:   41 89 f4 53 48 89 fb 48 83 ec 08 48 8b 07 4c 8b
0x00007fb68c49be58:   68 30 4d 85 ed 74 21 4c 89 ef e8 99 20 fc ff 48
0x00007fb68c49be68:   8b 03 4c 89 ef 44 01 60 78 48 83 c4 08 5b 41 5c
0x00007fb68c49be78:   41 5d 5d e9 b0 21 fc ff 01 70 78 48 83 c4 08 5b
0x00007fb68c49be88:   41 5c 41 5d 5d c3 66 90 48 8b 47 28 c3 90 66 2e
0x00007fb68c49be98:   0f 1f 84 00 00 00 00 00 48 83 c7 28 f0 48 0f c1
0x00007fb68c49bea8:   37 c3 66 0f 1f 44 00 00 48 8b 07 4c 8b 47 28 48
0x00007fb68c49beb8:   8b 40 38 48 85 c0 74 08 4c 89 c7 ff e0 0f 1f 00
0x00007fb68c49bec8:   c3 90 66 0f 1f 44 00 00 48 8b 07 c3 66 66 2e 0f
0x00007fb68c49bed8:   1f 84 00 00 00 00 00 90 55 48 89 e5 41 57 4c 8d
0x00007fb68c49bee8:   bf 38 02 00 00 41 56 49 89 d6 41 55 49 89 fd 41
0x00007fb68c49bef8:   54 49 89 f4 53 48 81 ec b8 00 00 00 64 48 8b 04
0x00007fb68c49bf08:   25 28 00 00 00 48 89 45 c8 31 c0 b8 01 00 00 00
0x00007fb68c49bf18:   f0 49 0f c1 07 48 8b 87 00 02 00 00 48 8d 8f 00
0x00007fb68c49bf28:   02 00 00 0f 1f 44 00 00 48 89 c2 48 89 c3 4c 31
0x00007fb68c49bf38:   e2 f0 48 0f b1 11 48 39 c3 75 ed 48 83 f8 ff 74
0x00007fb68c49bf48:   37 4c 39 e0 74 32 48 c7 c0 ff ff ff ff f0 49 0f
0x00007fb68c49bf58:   c1 07 48 8b 45 c8 64 48 2b 04 25 28 00 00 00 0f
0x00007fb68c49bf68:   85 88 01 00 00 48 81 c4 b8 00 00 00 5b 41 5c 41
0x00007fb68c49bf78:   5d 41 5e 41 5f 5d c3 90 48 8b 05 09 41 82 00 48
0x00007fb68c49bf88:   85 c0 0f 84 9c 00 00 00 4c 8d 8d 30 ff ff ff 4c
0x00007fb68c49bf98:   89 cf 4c 89 8d 28 ff ff ff e8 7a bc 01 00 48 8d
0x00007fb68c49bfa8:   05 3b 01 79 00 48 8d bd 60 ff ff ff 48 89 85 30
0x00007fb68c49bfb8:   ff ff ff e8 c0 93 ea ff 48 8d 05 99 40 82 00 48
0x00007fb68c49bfc8:   83 fb ff c7 45 b8 01 00 00 00 48 89 45 c0 48 8d
0x00007fb68c49bfd8:   05 43 01 79 00 4c 8b 8d 28 ff ff ff 48 89 85 30
0x00007fb68c49bfe8:   ff ff ff 0f 84 d0 00 00 00 49 8b 16 4c 89 cf 4c
0x00007fb68c49bff8:   89 e9 48 8d 35 ce 54 4a 00 31 c0 4c 89 8d 28 ff
0x00007fb68c49c008:   ff ff e8 d1 c5 01 00 4c 8b 8d 28 ff ff ff 48 8d 


Stack slot to memory mapping:

stack at sp + 0 slots: 0x0 is null
stack at sp + 1 slots: 0x00007fb68ccbcae0: <offset 0x00000000014ffae0> in /opt/javas/64/openjdk-21/lib/server/libjvm.so at 0x00007fb68b7bd000
stack at sp + 2 slots: 0x0 is null
stack at sp + 3 slots: 0x00000000000008be is an unknown value
stack at sp + 4 slots: 0x00007fb68ccbcae0: <offset 0x00000000014ffae0> in /opt/javas/64/openjdk-21/lib/server/libjvm.so at 0x00007fb68b7bd000
stack at sp + 5 slots: 0x0 is null
stack at sp + 6 slots: 0x00007fb68cefa050: <offset 0x000000000000f050> in /lib/x86_64-linux-gnu/libpthread.so.0 at 0x00007fb68ceeb000
stack at sp + 7 slots: 0x00007fb67852a960 is pointing into the stack for thread: 0x000055e7d1ec50f0


---------------  P R O C E S S  ---------------

Threads class SMR info:
_java_thread_list=0x000055e7d2ebedc0, length=107, elements={
0x000055e7d1ebd980, 0x000055e7d1ebf780, 0x000055e7d1ec0e10, 0x000055e7d1ec27d0,
0x000055e7d1ec50f0, 0x000055e7d1ec6b90, 0x000055e7d1ec90f0, 0x000055e7d20cb9c0,
0x000055e7d2341600, 0x000055e7d23f0390, 0x000055e7d2410cb0, 0x000055e7d2fc8650,
0x000055e7d2b6f5d0, 0x000055e7d241e5f0, 0x000055e7d2f071c0, 0x000055e7d25bda60,
0x000055e7d23d1c80, 0x000055e7d3315420, 0x000055e7d3316450, 0x000055e7d3458a00,
0x000055e7d345d340, 0x000055e7d345e670, 0x000055e7d3402af0, 0x000055e7d340a080,
0x000055e7d342b9c0, 0x000055e7d4b35f30, 0x000055e7d567a240, 0x000055e7d5603cd0,
0x000055e7d5e827a0, 0x000055e7d47fef00, 0x000055e7d4459cf0, 0x000055e7d835a6b0,
0x000055e7d64abe10, 0x000055e7d4b86f80, 0x000055e7d47fa090, 0x000055e7d47fb8f0,
0x000055e7d0aec380, 0x000055e7d31b9400, 0x000055e7d28249c0, 0x000055e7d817ecb0,
0x000055e7d69e6f20, 0x000055e7d69e7e50, 0x000055e7d87b1a80, 0x000055e7d8ff3790,
0x000055e7d3aeab20, 0x000055e7d4a1c730, 0x000055e7d8815950, 0x000055e7d82356c0,
0x000055e7d823a790, 0x000055e7d7e717c0, 0x000055e7d85ff230, 0x000055e7d4bd1a90,
0x000055e7d7889820, 0x000055e7d3330a40, 0x000055e7d8a81640, 0x000055e7d717f820,
0x000055e7d6102170, 0x000055e7d661bb40, 0x000055e7d660fcf0, 0x000055e7d6618190,
0x000055e7daa5ed80, 0x000055e7daa61230, 0x000055e7daa62f80, 0x000055e7dae455e0,
0x000055e7db929570, 0x000055e7dbbbc170, 0x000055e7dbd4f050, 0x000055e7dbffb0a0,
0x000055e7dc75a040, 0x000055e7dc90c7c0, 0x000055e7dbaa61a0, 0x000055e7db3abcb0,
0x000055e7dc2a27e0, 0x000055e7dc3a5de0, 0x000055e7dc2fc960, 0x000055e7d2170720,
0x000055e7d39b55b0, 0x000055e7d2b98fd0, 0x000055e7d702a400, 0x000055e7d3f492f0,
0x000055e7d3f4c600, 0x000055e7de9c9530, 0x000055e7de6b4cb0, 0x000055e7de6b6130,
0x000055e7de988a50, 0x000055e7dd79d850, 0x000055e7de991770, 0x000055e7de6acbc0,
0x000055e7de9bf9e0, 0x000055e7de5bf470, 0x000055e7e050eb20, 0x000055e7dc46eb30,
0x000055e7e0665b00, 0x000055e7e0926bd0, 0x000055e7deb5daf0, 0x000055e7e1a8ca80,
0x000055e7df280400, 0x000055e7d29522b0, 0x000055e7dda6c150, 0x000055e7d3344e50,
0x000055e7d31f8f00, 0x000055e7d86aa330, 0x000055e7dc421b60, 0x000055e7e230feb0,
0x000055e7e26d17f0, 0x000055e7df7a44f0, 0x000055e7e1083090
}

Java Threads: ( => current thread )
  0x000055e7d1ebd980 JavaThread "Reference Handler"          daemon [_thread_blocked, id=407739, stack(0x00007fb6785ef000,0x00007fb678630000) (260K)]
  0x000055e7d1ebf780 JavaThread "Finalizer"                  daemon [_thread_blocked, id=407740, stack(0x00007fb6785ae000,0x00007fb6785ef000) (260K)]
  0x000055e7d1ec0e10 JavaThread "Signal Dispatcher"          daemon [_thread_blocked, id=407741, stack(0x00007fb67856d000,0x00007fb6785ae000) (260K)]
  0x000055e7d1ec27d0 JavaThread "Service Thread"             daemon [_thread_blocked, id=407742, stack(0x00007fb67852c000,0x00007fb67856d000) (260K)]
=>0x000055e7d1ec50f0 JavaThread "Monitor Deflation Thread"   daemon [_thread_in_vm, id=407743, stack(0x00007fb6784eb000,0x00007fb67852c000) (260K)]
  0x000055e7d1ec6b90 JavaThread "C2 CompilerThread0"         daemon [_thread_blocked, id=407744, stack(0x00007fb6783ea000,0x00007fb6784eb000) (1028K)]
  0x000055e7d1ec90f0 JavaThread "C1 CompilerThread0"         daemon [_thread_blocked, id=407745, stack(0x00007fb6782e9000,0x00007fb6783ea000) (1028K)]
  0x000055e7d20cb9c0 JavaThread "Common-Cleaner"             daemon [_thread_blocked, id=407754, stack(0x00007fb6781d7000,0x00007fb678218000) (260K)]
  0x000055e7d2341600 JavaThread "Notification Thread"        daemon [_thread_blocked, id=407759, stack(0x00007fb67807d000,0x00007fb6780be000) (260K)]
  0x000055e7d23f0390 JavaThread "rebel-perf-logger"          daemon [_thread_blocked, id=407768, stack(0x00007fb673cf7000,0x00007fb673d38000) (260K)]
  0x000055e7d2410cb0 JavaThread "rebel-CacheKeepAlive"       daemon [_thread_blocked, id=407769, stack(0x00007fb673cb6000,0x00007fb673cf7000) (260K)]
  0x000055e7d2fc8650 JavaThread "process reaper (pid 407795)" daemon [_thread_in_native, id=407789, stack(0x00007fb67801a000,0x00007fb67803c000) (136K)]
  0x000055e7d2b6f5d0 JavaThread "rebel-logger"               daemon [_thread_blocked, id=407794, stack(0x00007fb673366000,0x00007fb6733a7000) (260K)]
  0x000055e7d241e5f0 JavaThread "rebel-fsnotify-ShutdownOnTermination" daemon [_thread_blocked, id=407796, stack(0x00007fb673c34000,0x00007fb673c75000) (260K)]
  0x000055e7d2f071c0 JavaThread "rebel-fsnotify-OutputReader" daemon [_thread_in_native, id=407797, stack(0x00007fb673c75000,0x00007fb673cb6000) (260K)]
  0x000055e7d25bda60 JavaThread "rebel-fsnotify-OutputReader" daemon [_thread_in_native, id=407798, stack(0x00007fb673aae000,0x00007fb673aef000) (260K)]
  0x000055e7d23d1c80 JavaThread "rebel-cache-writer"         daemon [_thread_blocked, id=407801, stack(0x00007fb673d38000,0x00007fb673d79000) (260K)]
  0x000055e7d3315420 JavaThread "rebel-weak-reaper"          daemon [_thread_blocked, id=407804, stack(0x00007fb673d79000,0x00007fb673dba000) (260K)]
  0x000055e7d3316450 JavaThread "rebel-jitter-thread"        daemon [_thread_blocked, id=407805, stack(0x00007fb673e7d000,0x00007fb673ebe000) (260K)]
  0x000055e7d3458a00 JavaThread "rebel-leaseManager-1"       daemon [_thread_blocked, id=407816, stack(0x00007fb673ebe000,0x00007fb673eff000) (260K)]
  0x000055e7d345d340 JavaThread "rebel-redeploy-thread"      daemon [_thread_blocked, id=407818, stack(0x00007fb673dba000,0x00007fb673dfb000) (260K)]
  0x000055e7d345e670 JavaThread "rebel-heartbeat-thread"     daemon [_thread_blocked, id=407819, stack(0x00007fb673dfb000,0x00007fb673e3c000) (260K)]
  0x000055e7d3402af0 JavaThread "rebel-debugger-attach-notifier" daemon [_thread_blocked, id=407820, stack(0x00007fb67803c000,0x00007fb67807d000) (260K)]
  0x000055e7d340a080 JavaThread "rebel-debugger-thread"      daemon [_thread_blocked, id=407821, stack(0x00007fb673bf3000,0x00007fb673c34000) (260K)]
  0x000055e7d342b9c0 JavaThread "rebel-change-detector-thread" daemon [_thread_blocked, id=407822, stack(0x00007fb673bb2000,0x00007fb673bf3000) (260K)]
  0x000055e7d4b35f30 JavaThread "Reference Reaper"           daemon [_thread_blocked, id=407875, stack(0x00007fb673180000,0x00007fb6731c1000) (260K)]
  0x000055e7d567a240 JavaThread "rebel-messaging-executor-83" daemon [_thread_blocked, id=408022, stack(0x00007fb673aef000,0x00007fb673b30000) (260K)]
  0x000055e7d5603cd0 JavaThread "rebel-IDENotificationsImpl-PostCycle" daemon [_thread_blocked, id=408027, stack(0x00007fb673b71000,0x00007fb673bb2000) (260K)]
  0x000055e7d5e827a0 JavaThread "MSC service thread 1-1"            [_thread_blocked, id=408143, stack(0x00007fb673b30000,0x00007fb673b71000) (260K)]
  0x000055e7d47fef00 JavaThread "MSC service thread 1-2"            [_thread_blocked, id=408146, stack(0x00007fb673e3c000,0x00007fb673e7d000) (260K)]
  0x000055e7d4459cf0 JavaThread "MSC service thread 1-3"            [_thread_blocked, id=408147, stack(0x00007fb673224000,0x00007fb673265000) (260K)]
  0x000055e7d835a6b0 JavaThread "MSC service thread 1-4"            [_thread_blocked, id=408156, stack(0x00007fb67313f000,0x00007fb673180000) (260K)]
  0x000055e7d64abe10 JavaThread "MSC service thread 1-5"            [_thread_blocked, id=408157, stack(0x00007fb6731c1000,0x00007fb673202000) (260K)]
  0x000055e7d4b86f80 JavaThread "MSC service thread 1-6"            [_thread_blocked, id=408158, stack(0x00007fb672db6000,0x00007fb672df7000) (260K)]
  0x000055e7d47fa090 JavaThread "MSC service thread 1-7"            [_thread_blocked, id=408159, stack(0x00007fb672d75000,0x00007fb672db6000) (260K)]
  0x000055e7d47fb8f0 JavaThread "MSC service thread 1-8"            [_thread_blocked, id=408160, stack(0x00007fb672d34000,0x00007fb672d75000) (260K)]
  0x000055e7d0aec380 JavaThread "DestroyJavaVM"                     [_thread_blocked, id=407380, stack(0x00007fb68b623000,0x00007fb68b664000) (260K)]
  0x000055e7d31b9400 JavaThread "ServerService Thread Pool -- 1"        [_thread_blocked, id=408183, stack(0x00007fb672ec9000,0x00007fb672f0a000) (260K)]
  0x000055e7d28249c0 JavaThread "DeploymentScanner-threads - 1"        [_thread_blocked, id=408267, stack(0x00007fb672410000,0x00007fb672451000) (260K)]
  0x000055e7d817ecb0 JavaThread "ServerService Thread Pool -- 43"        [_thread_blocked, id=408272, stack(0x00007fb67230c000,0x00007fb67234d000) (260K)]
  0x000055e7d69e6f20 JavaThread "management I/O-1"                  [_thread_in_native, id=408278, stack(0x00007fb6722cb000,0x00007fb67230c000) (260K)]
  0x000055e7d69e7e50 JavaThread "management I/O-2"                  [_thread_in_native, id=408279, stack(0x00007fb67228a000,0x00007fb6722cb000) (260K)]
  0x000055e7d87b1a80 JavaThread "management Accept"                 [_thread_in_native, id=408280, stack(0x00007fb672249000,0x00007fb67228a000) (260K)]
  0x000055e7d8ff3790 JavaThread "Timer-0"                           [_thread_blocked, id=408322, stack(0x00007fb66d862000,0x00007fb66d8a3000) (260K)]
  0x000055e7d3aeab20 JavaThread "default I/O-1"                     [_thread_in_native, id=408323, stack(0x00007fb66d821000,0x00007fb66d862000) (260K)]
  0x000055e7d4a1c730 JavaThread "default I/O-2"                     [_thread_in_native, id=408324, stack(0x00007fb66d7e0000,0x00007fb66d821000) (260K)]
  0x000055e7d8815950 JavaThread "default I/O-3"                     [_thread_in_native, id=408325, stack(0x00007fb66d79f000,0x00007fb66d7e0000) (260K)]
  0x000055e7d82356c0 JavaThread "default I/O-4"                     [_thread_in_native, id=408327, stack(0x00007fb66d75e000,0x00007fb66d79f000) (260K)]
  0x000055e7d823a790 JavaThread "default I/O-5"                     [_thread_in_native, id=408328, stack(0x00007fb66d71d000,0x00007fb66d75e000) (260K)]
  0x000055e7d7e717c0 JavaThread "default I/O-6"                     [_thread_in_native, id=408329, stack(0x00007fb66d6dc000,0x00007fb66d71d000) (260K)]
  0x000055e7d85ff230 JavaThread "default I/O-7"                     [_thread_in_native, id=408330, stack(0x00007fb66d69b000,0x00007fb66d6dc000) (260K)]
  0x000055e7d4bd1a90 JavaThread "default I/O-8"                     [_thread_in_native, id=408331, stack(0x00007fb66d65a000,0x00007fb66d69b000) (260K)]
  0x000055e7d7889820 JavaThread "default I/O-9"                     [_thread_in_native, id=408332, stack(0x00007fb66d619000,0x00007fb66d65a000) (260K)]
  0x000055e7d3330a40 JavaThread "ConnectionValidator"        daemon [_thread_blocked, id=408333, stack(0x00007fb66d5d8000,0x00007fb66d619000) (260K)]
  0x000055e7d8a81640 JavaThread "IdleRemover"                daemon [_thread_blocked, id=408334, stack(0x00007fb66d597000,0x00007fb66d5d8000) (260K)]
  0x000055e7d717f820 JavaThread "default I/O-10"                    [_thread_in_native, id=408335, stack(0x00007fb66d556000,0x00007fb66d597000) (260K)]
  0x000055e7d6102170 JavaThread "default I/O-11"                    [_thread_in_native, id=408336, stack(0x00007fb66d515000,0x00007fb66d556000) (260K)]
  0x000055e7d661bb40 JavaThread "default I/O-12"                    [_thread_in_native, id=408337, stack(0x00007fb66d4d4000,0x00007fb66d515000) (260K)]
  0x000055e7d660fcf0 JavaThread "default I/O-13"                    [_thread_in_native, id=408338, stack(0x00007fb66d493000,0x00007fb66d4d4000) (260K)]
  0x000055e7d6618190 JavaThread "default I/O-14"                    [_thread_in_native, id=408339, stack(0x00007fb66d452000,0x00007fb66d493000) (260K)]
  0x000055e7daa5ed80 JavaThread "default I/O-15"                    [_thread_in_native, id=408340, stack(0x00007fb66d411000,0x00007fb66d452000) (260K)]
  0x000055e7daa61230 JavaThread "default I/O-16"                    [_thread_in_native, id=408341, stack(0x00007fb66d3d0000,0x00007fb66d411000) (260K)]
  0x000055e7daa62f80 JavaThread "default Accept"                    [_thread_in_native, id=408342, stack(0x00007fb66d38f000,0x00007fb66d3d0000) (260K)]
  0x000055e7dae455e0 JavaThread "mysql-cj-abandoned-connection-cleanup" daemon [_thread_blocked, id=408343, stack(0x00007fb66d34e000,0x00007fb66d38f000) (260K)]
  0x000055e7db929570 JavaThread "Timer-1"                           [_thread_blocked, id=408344, stack(0x00007fb66d30d000,0x00007fb66d34e000) (260K)]
  0x000055e7dbbbc170 JavaThread "XNIO-1 I/O-1"               daemon [_thread_in_native, id=408350, stack(0x00007fb66d2cc000,0x00007fb66d30d000) (260K)]
  0x000055e7dbd4f050 JavaThread "XNIO-1 Accept"              daemon [_thread_in_native, id=408351, stack(0x00007fb66d28b000,0x00007fb66d2cc000) (260K)]
  0x000055e7dbffb0a0 JavaThread "Transaction Expired Entry Monitor" daemon [_thread_blocked, id=408370, stack(0x00007fb66d24a000,0x00007fb66d28b000) (260K)]
  0x000055e7dc75a040 JavaThread "Periodic Recovery"                 [_thread_blocked, id=408371, stack(0x00007fb66d209000,0x00007fb66d24a000) (260K)]
  0x000055e7dc90c7c0 JavaThread "Transaction Reaper"         daemon [_thread_blocked, id=408373, stack(0x00007fb66d1c8000,0x00007fb66d209000) (260K)]
  0x000055e7dbaa61a0 JavaThread "Transaction Reaper Worker 0" daemon [_thread_blocked, id=408374, stack(0x00007fb66d187000,0x00007fb66d1c8000) (260K)]
  0x000055e7db3abcb0 JavaThread "DeploymentScanner-threads - 2"        [_thread_blocked, id=408375, stack(0x00007fb66d146000,0x00007fb66d187000) (260K)]
  0x000055e7dc2a27e0 JavaThread "DeploymentScanner-threads - 3"        [_thread_blocked, id=408376, stack(0x00007fb66d105000,0x00007fb66d146000) (260K)]
  0x000055e7dc3a5de0 JavaThread "ORB Run Thread"                    [_thread_blocked, id=408385, stack(0x00007fb66d0c4000,0x00007fb66d105000) (260K)]
  0x000055e7dc2fc960 JavaThread "SelectorThread"             daemon [_thread_in_native, id=408387, stack(0x00007fb66d083000,0x00007fb66d0c4000) (260K)]
  0x000055e7d2170720 JavaThread "ServerService Thread Pool -- 83"        [_thread_blocked, id=408663, stack(0x00007fb66cd6b000,0x00007fb66cdac000) (260K)]
  0x000055e7d39b55b0 JavaThread "default task-1"                    [_thread_blocked, id=408782, stack(0x00007fb66cd2a000,0x00007fb66cd6b000) (260K)]
  0x000055e7d2b98fd0 JavaThread "ServerService Thread Pool -- 84"        [_thread_blocked, id=409159, stack(0x00007fb66c794000,0x00007fb66c7d5000) (260K)]
  0x000055e7d702a400 JavaThread "Reference Reaper #1"        daemon [_thread_blocked, id=409230, stack(0x00007fb66c816000,0x00007fb66c857000) (260K)]
  0x000055e7d3f492f0 JavaThread "Reference Reaper #2"        daemon [_thread_blocked, id=409231, stack(0x00007fb66c7d5000,0x00007fb66c816000) (260K)]
  0x000055e7d3f4c600 JavaThread "Reference Reaper #3"        daemon [_thread_blocked, id=409232, stack(0x00007fb66d042000,0x00007fb66d083000) (260K)]
  0x000055e7de9c9530 JavaThread "Weld Thread Pool -- 1"             [_thread_blocked, id=409462, stack(0x00007fb672b2c000,0x00007fb672b6d000) (260K)]
  0x000055e7de6b4cb0 JavaThread "Weld Thread Pool -- 2"             [_thread_blocked, id=409463, stack(0x00007fb66d925000,0x00007fb66d966000) (260K)]
  0x000055e7de6b6130 JavaThread "Weld Thread Pool -- 9"             [_thread_blocked, id=409464, stack(0x00007fb6725d7000,0x00007fb672618000) (260K)]
  0x000055e7de988a50 JavaThread "Weld Thread Pool -- 4"             [_thread_blocked, id=409465, stack(0x00007fb6724d3000,0x00007fb672514000) (260K)]
  0x000055e7dd79d850 JavaThread "Weld Thread Pool -- 5"             [_thread_blocked, id=409466, stack(0x00007fb67279e000,0x00007fb6727df000) (260K)]
  0x000055e7de991770 JavaThread "Weld Thread Pool -- 6"             [_thread_blocked, id=409467, stack(0x00007fb66d8e4000,0x00007fb66d925000) (260K)]
  0x000055e7de6acbc0 JavaThread "Weld Thread Pool -- 7"             [_thread_blocked, id=409468, stack(0x00007fb672b6d000,0x00007fb672bae000) (260K)]
  0x000055e7de9bf9e0 JavaThread "Weld Thread Pool -- 8"             [_thread_blocked, id=409469, stack(0x00007fb672cb2000,0x00007fb672cf3000) (260K)]
  0x000055e7de5bf470 JavaThread "Weld Thread Pool -- 3"             [_thread_blocked, id=409470, stack(0x00007fb672451000,0x00007fb672492000) (260K)]
  0x000055e7e050eb20 JavaThread "ServerService Thread Pool -- 99"        [_thread_blocked, id=414294, stack(0x00007fb672492000,0x00007fb6724d3000) (260K)]
  0x000055e7dc46eb30 JavaThread "EJB default - 1"                   [_thread_blocked, id=414857, stack(0x00007fb6729a6000,0x00007fb6729e7000) (260K)]
  0x000055e7e0665b00 JavaThread "EJB default - 2"                   [_thread_blocked, id=414858, stack(0x00007fb672aeb000,0x00007fb672b2c000) (260K)]
  0x000055e7e0926bd0 JavaThread "EJB default - 3"                   [_thread_blocked, id=414859, stack(0x00007fb672965000,0x00007fb6729a6000) (260K)]
  0x000055e7deb5daf0 JavaThread "rebel/pool-1-thread-1"      daemon [_thread_blocked, id=415099, stack(0x00007fb672041000,0x00007fb672082000) (260K)]
  0x000055e7e1a8ca80 JavaThread "EJB default - 4"                   [_thread_blocked, id=415181, stack(0x00007fb66debb000,0x00007fb66defc000) (260K)]
  0x000055e7df280400 JavaThread "EJB default - 5"                   [_thread_blocked, id=415362, stack(0x00007fb672618000,0x00007fb672659000) (260K)]
  0x000055e7d29522b0 JavaThread "ForkJoinPool.commonPool-worker-1" daemon [_thread_blocked, id=415474, stack(0x00007fb66da29000,0x00007fb66da6a000) (260K)]
  0x000055e7dda6c150 JavaThread "ForkJoinPool.commonPool-worker-2" daemon [_thread_blocked, id=415475, stack(0x00007fb672514000,0x00007fb672555000) (260K)]
  0x000055e7d3344e50 JavaThread "ForkJoinPool.commonPool-worker-3" daemon [_thread_blocked, id=415476, stack(0x00007fb672186000,0x00007fb6721c7000) (260K)]
  0x000055e7d31f8f00 JavaThread "ForkJoinPool.commonPool-worker-6" daemon [_thread_blocked, id=415477, stack(0x00007fb66dcb3000,0x00007fb66dcf4000) (260K)]
  0x000055e7d86aa330 JavaThread "ForkJoinPool.commonPool-worker-5" daemon [_thread_blocked, id=415478, stack(0x00007fb672c30000,0x00007fb672c71000) (260K)]
  0x000055e7dc421b60 JavaThread "ForkJoinPool.commonPool-worker-7" daemon [_thread_blocked, id=415480, stack(0x00007fb67275d000,0x00007fb67279e000) (260K)]
  0x000055e7e230feb0 JavaThread "C2 CompilerThread1"         daemon [_thread_blocked, id=420933, stack(0x00007fb673eff000,0x00007fb674000000) (1028K)]
  0x000055e7e26d17f0 JavaThread "ServerService Thread Pool -- 103"        [_thread_blocked, id=420979, stack(0x00007fb66dd35000,0x00007fb66dd76000) (260K)]
  0x000055e7df7a44f0 JavaThread "default task-2"                    [_thread_blocked, id=420995, stack(0x00007fb66dbaf000,0x00007fb66dbf0000) (260K)]
  0x000055e7e1083090 JavaThread "C2 CompilerThread2"         daemon [_thread_blocked, id=421033, stack(0x00007fb6780be000,0x00007fb6781bf000) (1028K)]
Total: 107

Other Threads:
  0x000055e7d1dd71e0 VMThread "VM Thread"                           [id=407738, stack(0x00007fb678918000,0x00007fb678a18000) (1024K)] _threads_hazard_ptr=0x000055e7d2ebedc0
  0x000055e7d1d8c390 WatcherThread "VM Periodic Task Thread"        [id=407735, stack(0x00007fb678a19000,0x00007fb678b19000) (1024K)]
  0x000055e7d1363550 WorkerThread "GC Thread#0"                     [id=407730, stack(0x00007fb67af1e000,0x00007fb67b01e000) (1024K)]
  0x000055e7d273a520 WorkerThread "GC Thread#1"                     [id=407781, stack(0x00007fb6739ae000,0x00007fb673aae000) (1024K)]
  0x000055e7d26204a0 WorkerThread "GC Thread#2"                     [id=407782, stack(0x00007fb6738ad000,0x00007fb6739ad000) (1024K)]
  0x000055e7d26292b0 WorkerThread "GC Thread#3"                     [id=407783, stack(0x00007fb6737ac000,0x00007fb6738ac000) (1024K)]
  0x000055e7d262a0f0 WorkerThread "GC Thread#4"                     [id=407784, stack(0x00007fb6736ab000,0x00007fb6737ab000) (1024K)]
  0x000055e7d262af50 WorkerThread "GC Thread#5"                     [id=407785, stack(0x00007fb6735aa000,0x00007fb6736aa000) (1024K)]
  0x000055e7d262bde0 WorkerThread "GC Thread#6"                     [id=407786, stack(0x00007fb6734a9000,0x00007fb6735a9000) (1024K)]
  0x000055e7d262d060 WorkerThread "GC Thread#7"                     [id=407787, stack(0x00007fb6733a8000,0x00007fb6734a8000) (1024K)]
  0x000055e7d136f420 ConcurrentGCThread "G1 Main Marker"            [id=407731, stack(0x00007fb67ae1d000,0x00007fb67af1d000) (1024K)]
  0x000055e7d1370700 WorkerThread "G1 Conc#0"                       [id=407732, stack(0x00007fb67ad1c000,0x00007fb67ae1c000) (1024K)]
  0x000055e7d2745d20 WorkerThread "G1 Conc#1"                       [id=407806, stack(0x00007fb673266000,0x00007fb673366000) (1024K)]
  0x000055e7d1c61820 ConcurrentGCThread "G1 Refine#0"               [id=407733, stack(0x00007fb678c1b000,0x00007fb678d1b000) (1024K)]
  0x000055e7d1c62b10 ConcurrentGCThread "G1 Service"                [id=407734, stack(0x00007fb678b1a000,0x00007fb678c1a000) (1024K)]
Total: 15

Threads with active compile tasks:
C2 CompilerThread0   308663 46834       4       org.jboss.weld.util.reflection.HierarchyDiscovery::discoverInterfaces (77 bytes)
C1 CompilerThread0   308664 48275       2       java.lang.management.ThreadInfo::initialize (207 bytes)
C2 CompilerThread1   308664 47646 % !   4       com.zeroturnaround.jrebelbase.facade.l::filterMethods @ 53 (197 bytes)
C2 CompilerThread2   308664 45243 %     4       java.util.regex.Matcher::reset @ 17 (150 bytes)
Total: 4

VM state: synchronizing (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00007fb68ccbcda0] Threads_lock - owner thread: 0x000055e7d1dd71e0

Heap address: 0x00000000ad400000, size: 1324 MB, Compressed Oops mode: 32-bit

CDS archive(s) not mapped
Compressed class space mapped at: 0x0000000100000000-0x0000000140000000, reserved size: 1073741824
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3, Narrow klass range: 0x140000000

GC Precious Log:
 CardTable entry size: 512
 Card Set container configuration: InlinePtr #cards 5 size 8 Array Of Cards #cards 12 size 40 Howl #buckets 4 coarsen threshold 1843 Howl Bitmap #cards 512 size 80 coarsen threshold 460 Card regions per heap region 1 cards per card region 2048
 CPUs: 8 total, 8 available
 Memory: 15634M
 Large Page Support: Disabled
 NUMA Support: Disabled
 Compressed Oops: Enabled (32-bit)
 Heap Region Size: 1M
 Heap Min Capacity: 8M
 Heap Initial Capacity: 246M
 Heap Max Capacity: 1324M
 Pre-touch: Disabled
 Parallel Workers: 8
 Concurrent Workers: 2
 Concurrent Refinement Workers: 8
 Periodic GC: Disabled

Heap:
 garbage-first heap   total 1089536K, used 627674K [0x00000000ad400000, 0x0000000100000000)
  region size 1024K, 67 young (68608K), 18 survivors (18432K)
 Metaspace       used 194204K, committed 200256K, reserved 1245184K
  class space    used 21601K, committed 23744K, reserved 1048576K

Heap Regions: E=young(eden), S=young(survivor), O=old, HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, TAMS=top-at-mark-start, PB=parsable bottom
|   0|0x00000000ad400000, 0x00000000ad500000, 0x00000000ad500000|100%|HS|  |TAMS 0x00000000ad400000| PB 0x00000000ad400000| Complete 
...
|1063|0x00000000efb00000, 0x00000000efc00000, 0x00000000efc00000|100%| E|CS|TAMS 0x00000000efb00000| PB 0x00000000efb00000| Complete 

Card table byte_map: [0x00007fb67c524000,0x00007fb67c7ba000] _byte_map_base: 0x00007fb67bfba000

Marking Bits: (CMBitMap*) 0x000055e7d13643b0
 Bits: [0x00007fb67b074000, 0x00007fb67c524000)

Polling page: 0x00007fb68cf1c000

Metaspace:

Usage:
  Non-class:    168.56 MB used.
      Class:     21.09 MB used.
       Both:    189.65 MB used.

Virtual space:
  Non-class space:      192.00 MB reserved,     172.38 MB ( 90%) committed,  3 nodes.
      Class space:        1.00 GB reserved,      23.19 MB (  2%) committed,  1 nodes.
             Both:        1.19 GB reserved,     195.56 MB ( 16%) committed. 

Chunk freelists:
   Non-Class:  2.77 MB
       Class:  8.67 MB
        Both:  11.44 MB

MaxMetaspaceSize: 1.50 GB
CompressedClassSpaceSize: 1.00 GB
Initial GC threshold: 21.00 MB
Current GC threshold: 325.75 MB
CDS: off
 - commit_granule_bytes: 65536.
 - commit_granule_words: 8192.
 - virtual_space_node_default_size: 8388608.
 - enlarge_chunks_in_place: 1.
 - use_allocation_guard: 0.


Internal statistics:

num_allocs_failed_limit: 30.
num_arena_births: 2560.
num_arena_deaths: 26.
num_vsnodes_births: 4.
num_vsnodes_deaths: 0.
num_space_committed: 3128.
num_space_uncommitted: 0.
num_chunks_returned_to_freelist: 59.
num_chunks_taken_from_freelist: 11587.
num_chunk_merges: 22.
num_chunk_splits: 6898.
num_chunks_enlarged: 3384.
num_inconsistent_stats: 0.

CodeCache: size=98304Kb used=24565Kb max_used=66298Kb free=73738Kb
 bounds [0x00007fb67cb10000, 0x00007fb680be0000, 0x00007fb682b10000]
 total_blobs=13848 nmethods=12942 adapters=809
 compilation: enabled
              stopped_count=0, restarted_count=0
 full_count=0

Compilation events (20 events):
Event: 307.680 Thread 0x000055e7d1ec90f0 48265       2       com.zeroturnaround.javarebel.idz::remove (137 bytes)
Event: 307.686 Thread 0x000055e7d1ec90f0 nmethod 48265 0x00007fb67cdded90 code [0x00007fb67cddef80, 0x00007fb67cddf3e8]
Event: 307.686 Thread 0x000055e7d1ec90f0 48272       2       com.zeroturnaround.javarebel.hhx::<init> (31 bytes)
Event: 307.687 Thread 0x000055e7d1ec90f0 nmethod 48272 0x00007fb67ce77910 code [0x00007fb67ce77ac0, 0x00007fb67ce77cd0]
Event: 307.687 Thread 0x000055e7d1ec90f0 48269       3       com.zeroturnaround.javarebel.hxo::<init> (186 bytes)
Event: 307.687 Thread 0x000055e7d1ec90f0 nmethod 48269 0x00007fb67d4a1690 code [0x00007fb67d4a19a0, 0x00007fb67d4a2cf0]
Event: 307.687 Thread 0x000055e7d1ec90f0 48271       2       com.zeroturnaround.javarebel.hnx::a (75 bytes)
Event: 307.688 Thread 0x000055e7d1ec90f0 nmethod 48271 0x00007fb67cf18f10 code [0x00007fb67cf19100, 0x00007fb67cf19408]
Event: 307.688 Thread 0x000055e7d1ec90f0 48268       2       com.zeroturnaround.javarebel.hmd::a (7 bytes)
Event: 307.688 Thread 0x000055e7d1ec90f0 nmethod 48268 0x00007fb67d23cf10 code [0x00007fb67d23d0c0, 0x00007fb67d23d280]
Event: 307.693 Thread 0x000055e7e1083090 nmethod 45704% 0x00007fb67eebff10 code [0x00007fb67eec0380, 0x00007fb67eec2838]
Event: 307.694 Thread 0x000055e7e1083090 48266       4       com.zeroturnaround.javarebel.heu::a (161 bytes)
Event: 307.695 Thread 0x000055e7d1ec90f0 48273       3       com.zeroturnaround.javarebel.hfs::a (109 bytes)
Event: 307.695 Thread 0x000055e7d1ec90f0 nmethod 48273 0x00007fb67ce89990 code [0x00007fb67ce89ba0, 0x00007fb67ce8a0d8]
Event: 307.699 Thread 0x000055e7d1ec6b90 nmethod 44405 0x00007fb67cd3ca10 code [0x00007fb67cd3ccc0, 0x00007fb67cd3da90]
Event: 307.699 Thread 0x000055e7d1ec6b90 47571       4       com.zeroturnaround.javarebel.hhh::a (60 bytes)
Event: 307.700 Thread 0x000055e7d1ec90f0 48274       2       java.nio.channels.spi.AbstractInterruptibleChannel::_jr$ig$interruptor (8 bytes)
Event: 307.700 Thread 0x000055e7d1ec90f0 nmethod 48274 0x00007fb67d275510 code [0x00007fb67d2756a0, 0x00007fb67d2757b8]
Event: 307.703 Thread 0x000055e7e1083090 nmethod 48266 0x00007fb67cd40510 code [0x00007fb67cd40700, 0x00007fb67cd41058]
Event: 307.703 Thread 0x000055e7e1083090 45243 %     4       java.util.regex.Matcher::reset @ 17 (150 bytes)

GC Heap History (20 events):
Event: 211.167 GC heap before
{Heap before GC invocations=104 (full 0):
 garbage-first heap   total 829440K, used 698065K [0x00000000ad400000, 0x0000000100000000)
  region size 1024K, 244 young (249856K), 10 survivors (10240K)
 Metaspace       used 185336K, committed 190784K, reserved 1245184K
  class space    used 20673K, committed 22656K, reserved 1048576K
}
Event: 211.218 GC heap after
{Heap after GC invocations=105 (full 0):
 garbage-first heap   total 829440K, used 466431K [0x00000000ad400000, 0x0000000100000000)
  region size 1024K, 17 young (17408K), 17 survivors (17408K)
 Metaspace       used 185336K, committed 190784K, reserved 1245184K
  class space    used 20673K, committed 22656K, reserved 1048576K
}
Event: 216.924 GC heap before
{Heap before GC invocations=105 (full 0):
 garbage-first heap   total 829440K, used 768511K [0x00000000ad400000, 0x0000000100000000)
  region size 1024K, 313 young (320512K), 17 survivors (17408K)
 Metaspace       used 185694K, committed 191168K, reserved 1245184K
  class space    used 20694K, committed 22656K, reserved 1048576K
}
Event: 216.957 GC heap after
{Heap after GC invocations=106 (full 0):
 garbage-first heap   total 829440K, used 479735K [0x00000000ad400000, 0x0000000100000000)
  region size 1024K, 31 young (31744K), 31 survivors (31744K)
 Metaspace       used 185694K, committed 191168K, reserved 1245184K
  class space    used 20694K, committed 22656K, reserved 1048576K
}
Event: 218.428 GC heap before
{Heap before GC invocations=106 (full 0):
 garbage-first heap   total 829440K, used 754167K [0x00000000ad400000, 0x0000000100000000)
  region size 1024K, 300 young (307200K), 31 survivors (31744K)
 Metaspace       used 185919K, committed 191424K, reserved 1245184K
  class space    used 20720K, committed 22720K, reserved 1048576K
}

Dll operation events (12 events):
...

Deoptimization events (20 events):
Event: 307.365 Thread 0x000055e7e0665b00 DEOPT PACKING pc=0x00007fb67cc252f8 sp=0x00007fb672b291f0
Event: 307.365 Thread 0x000055e7e0665b00 DEOPT UNPACKING pc=0x00007fb67cb63f99 sp=0x00007fb672b290b8 mode 2
Event: 307.481 Thread 0x000055e7db3abcb0 DEOPT PACKING pc=0x00007fb680814660 sp=0x00007fb66d183bc0
Event: 307.481 Thread 0x000055e7db3abcb0 DEOPT UNPACKING pc=0x00007fb67cb6472f sp=0x00007fb66d183100 mode 0
Event: 307.556 Thread 0x000055e7dc2a27e0 DEOPT PACKING pc=0x00007fb67defeeac sp=0x00007fb66d144550
Event: 307.556 Thread 0x000055e7dc2a27e0 DEOPT UNPACKING pc=0x00007fb67cb6472f sp=0x00007fb66d143a20 mode 0
Event: 307.556 Thread 0x000055e7dc2a27e0 DEOPT PACKING pc=0x00007fb67e5228ac sp=0x00007fb66d1445a0
Event: 307.556 Thread 0x000055e7dc2a27e0 DEOPT UNPACKING pc=0x00007fb67cb6472f sp=0x00007fb66d143a60 mode 0
Event: 307.556 Thread 0x000055e7dc2a27e0 DEOPT PACKING pc=0x00007fb67e11774c sp=0x00007fb66d1445f0
Event: 307.556 Thread 0x000055e7dc2a27e0 DEOPT UNPACKING pc=0x00007fb67cb6472f sp=0x00007fb66d143b38 mode 0
Event: 307.629 Thread 0x000055e7d342b9c0 Uncommon trap: trap_request=0xffffffc6 fr.pc=0x00007fb67d328d94 relative=0x00000000000000d4
Event: 307.629 Thread 0x000055e7d342b9c0 Uncommon trap: reason=bimorphic_or_optimized_type_check action=maybe_recompile pc=0x00007fb67d328d94 method=java.util.Collections$UnmodifiableList$1.hasNext()Z @ 8 c2
Event: 307.629 Thread 0x000055e7d342b9c0 DEOPT PACKING pc=0x00007fb67d328d94 sp=0x00007fb673bf14f0
Event: 307.629 Thread 0x000055e7d342b9c0 DEOPT UNPACKING pc=0x00007fb67cb63f99 sp=0x00007fb673bf14b0 mode 2
Event: 307.629 Thread 0x000055e7d342b9c0 Uncommon trap: trap_request=0xffffffc6 fr.pc=0x00007fb67d328d94 relative=0x00000000000000d4
Event: 307.629 Thread 0x000055e7d342b9c0 Uncommon trap: reason=bimorphic_or_optimized_type_check action=maybe_recompile pc=0x00007fb67d328d94 method=java.util.Collections$UnmodifiableList$1.hasNext()Z @ 8 c2
Event: 307.629 Thread 0x000055e7d342b9c0 DEOPT PACKING pc=0x00007fb67d328d94 sp=0x00007fb673bf14f0
Event: 307.629 Thread 0x000055e7d342b9c0 DEOPT UNPACKING pc=0x00007fb67cb63f99 sp=0x00007fb673bf14b0 mode 2
Event: 307.671 Thread 0x000055e7d342b9c0 DEOPT PACKING pc=0x00007fb68094c3c3 sp=0x00007fb673bf0060
Event: 307.671 Thread 0x000055e7d342b9c0 DEOPT UNPACKING pc=0x00007fb67cb6472f sp=0x00007fb673bef4f0 mode 0

Classes loaded (20 events):
Event: 303.929 Loading class com/zeroturnaround/javarebel/acv
Event: 303.929 Loading class com/zeroturnaround/javarebel/acv done
Event: 303.931 Loading class com/zeroturnaround/javarebel/iai
Event: 303.931 Loading class com/zeroturnaround/javarebel/iai done
Event: 303.931 Loading class com/zeroturnaround/javarebel/htq
Event: 303.931 Loading class com/zeroturnaround/javarebel/htq done
Event: 303.932 Loading class jdk/internal/org/objectweb/asm/ClassReader
Event: 304.152 Loading class jdk/internal/org/objectweb/asm/ClassReader done
Event: 304.153 Loading class com/zeroturnaround/javarebel/hvp
Event: 304.153 Loading class com/zeroturnaround/javarebel/hvp done
Event: 304.172 Loading class com/zeroturnaround/javarebel/iaf
Event: 304.173 Loading class com/zeroturnaround/javarebel/iaf done
Event: 304.173 Loading class com/zeroturnaround/javarebel/iad
Event: 304.179 Loading class com/zeroturnaround/javarebel/iad done
Event: 304.179 Loading class com/zeroturnaround/javarebel/iae
Event: 304.179 Loading class com/zeroturnaround/javarebel/iae done
Event: 304.525 Loading class com/zeroturnaround/javarebel/bl
Event: 304.526 Loading class org/zeroturnaround/javarebel/StatisticsCollector$RedeployStatisticsEvent
Event: 304.526 Loading class org/zeroturnaround/javarebel/StatisticsCollector$RedeployStatisticsEvent done
Event: 304.526 Loading class com/zeroturnaround/javarebel/bl done

Classes unloaded (13 events):
Event: 50.501 Thread 0x000055e7d1dd71e0 Unloading class 0x0000000101278000 'java/lang/invoke/LambdaForm$DMH+0x0000000101278000'
...

Classes redefined (20 events):
Event: 299.193 Thread 0x000055e7d1dd71e0 redefined class name=com.zeroturnaround.jrebelbase.facade.e, count=2
Event: 299.193 Thread 0x000055e7d1dd71e0 redefined class name=com.zeroturnaround.jrebelbase.facade.g, count=2
Event: 299.193 Thread 0x000055e7d1dd71e0 redefined class name=com.zeroturnaround.jrebelbase.facade.ac, count=2
Event: 299.747 Thread 0x000055e7d1dd71e0 redefined class name=org.jboss.weld.interceptor.util.proxy.TargetInstanceProxy, count=1
Event: 299.747 Thread 0x000055e7d1dd71e0 redefined class name=test.PhobosBean$Proxy$_$$_Weld$EnterpriseProxy$, count=2
Event: 299.747 Thread 0x000055e7d1dd71e0 redefined class name=org.jboss.as.ee.component.serialization.WriteReplaceInterface, count=1
Event: 299.747 Thread 0x000055e7d1dd71e0 redefined class name=org.jboss.weld.bean.proxy.ProxyObject, count=1
Event: 299.747 Thread 0x000055e7d1dd71e0 redefined class name=java.io.Serializable, count=1
Event: 299.747 Thread 0x000055e7d1dd71e0 redefined class name=org.jboss.weld.interceptor.proxy.LifecycleMixin, count=1
Event: 299.747 Thread 0x000055e7d1dd71e0 redefined class name=org.jboss.weld.module.ejb.EnterpriseBeanInstance, count=1
Event: 299.747 Thread 0x000055e7d1dd71e0 redefined class name=org.jboss.weld.proxy.WeldConstruct, count=1
Event: 299.747 Thread 0x000055e7d1dd71e0 redefined class name=org.jboss.as.ejb3.validator.EjbProxy, count=1
Event: 299.747 Thread 0x000055e7d1dd71e0 redefined class name=test.PhobosBean, count=2
Event: 299.747 Thread 0x000055e7d1dd71e0 redefined class name=test.PhobosBean$$$view162, count=3
Event: 307.750 Thread 0x000055e7d1dd71e0 redefined class name=test.PhobosBean$Proxy$_$$_Weld$EnterpriseProxy$, count=2
Event: 307.750 Thread 0x000055e7d1dd71e0 redefined class name=test.PhobosBean, count=2
Event: 307.750 Thread 0x000055e7d1dd71e0 redefined class name=test.DeimosBean$Proxy$_$$_Weld$EnterpriseProxy$, count=2
Event: 307.750 Thread 0x000055e7d1dd71e0 redefined class name=test.DeimosBean, count=2
Event: 307.750 Thread 0x000055e7d1dd71e0 redefined class name=test.PhobosBean$$$view168, count=3
Event: 307.750 Thread 0x000055e7d1dd71e0 redefined class name=test.DeimosBean$$$view167, count=3

Internal exceptions (20 events):
Event: 268.313 Thread 0x000055e7d5e827a0 Exception <a 'java/lang/ClassNotFoundException'{0x00000000e4e799b0}: javax/smartcardio/CardPermission> (0x00000000e4e799b0) 
thrown [src/hotspot/share/classfile/systemDictionary.cpp, line 312]
Event: 268.313 Thread 0x000055e7d5e827a0 Exception <a 'java/io/FileNotFoundException'{0x00000000e4e86750}> (0x00000000e4e86750) 
thrown [src/hotspot/share/prims/jni.cpp, line 520]
Event: 268.609 Thread 0x000055e7d5e827a0 Exception <a 'java/lang/ClassNotFoundException'{0x00000000deb0f560}: javax/smartcardio/CardPermission> (0x00000000deb0f560) 
thrown [src/hotspot/share/classfile/systemDictionary.cpp, line 312]
...

ZGC Phase Switch (0 events):
No events

VM Operations (20 events):
Event: 302.436 Executing VM operation: ICBufferFull done
Event: 303.237 Executing VM operation: ICBufferFull
Event: 303.237 Executing VM operation: ICBufferFull done
Event: 303.539 Executing VM operation: ICBufferFull
Event: 303.539 Executing VM operation: ICBufferFull done
Event: 303.963 Executing VM operation: ICBufferFull
Event: 303.964 Executing VM operation: ICBufferFull done
Event: 303.972 Executing VM operation: G1TryInitiateConcMark (CodeCache GC Threshold)
Event: 304.029 Executing VM operation: G1TryInitiateConcMark (CodeCache GC Threshold) done
Event: 305.029 Executing VM operation: Cleanup
Event: 305.040 Executing VM operation: Cleanup done
Event: 305.574 Executing VM operation: G1PauseRemark
Event: 305.738 Executing VM operation: G1PauseRemark done
Event: 306.510 Executing VM operation: G1PauseCleanup
Event: 306.512 Executing VM operation: G1PauseCleanup done
Event: 307.524 Executing VM operation: Cleanup
Event: 307.527 Executing VM operation: Cleanup done
Event: 307.707 Executing VM operation: RedefineClasses
Event: 307.846 Executing VM operation: RedefineClasses done
Event: 307.846 Executing VM operation: ThreadDump

Events (20 events):
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc3fa90
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc3c210
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc3ab90
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc39a10
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc47310
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc46b10
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc42310
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc41d10
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc41410
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc40d10
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc32890
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc30b90
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc2eb10
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc38b10
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc36f90
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc36810
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc2c210
Event: 305.714 Thread 0x000055e7d1dd71e0 flushing nmethod 0x00007fb67cc24d10
Event: 307.556 Thread 0x000055e7e1083090 Thread added: 0x000055e7e1083090
Event: 307.557 Protecting memory [0x00007fb6780be000,0x00007fb6780c2000] with protection modes 0


Dynamic libraries:
ad400000-efc00000 rw-p 00000000 00:00 0 
efc00000-100000000 ---p 00000000 00:00 0 
100000000-100040000 rw-p 00000000 00:00 0 
100040000-1000c0000 rw-p 00000000 00:00 0 
101750000-140000000 ---p 00000000 00:00 0 
55e7cf8a5000-55e7cf8a6000 r--p 00000000 103:01 4388662                   /opt/javas/64/openjdk-21/bin/java
...
7fb682b10000-7fb682c32000 r--s 0143c000 103:01 13986080                  /home/zeroturnaround/jenkins-agent/workspace/javeleon-ejbtest-wildfly-latest/2f606c00/tmp/jrebel/jrebel.jar
...
7fb682c99000-7fb682cbf000 r-xp 00000000 103:01 13986075                  /home/zeroturnaround/jenkins-agent/workspace/javeleon-ejbtest-wildfly-latest/2f606c00/tmp/jrebel/lib/libjrebel64.so
...
7ffe929ec000-7ffe929ee000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]
Total number of mappings: 661


VM Arguments:
jvm_args: -D[Standalone] -Djdk.serialFilter=maxbytes=10485760;maxdepth=128;maxarray=100000;maxrefs=300000 -DIS_CYGWIN=false -agentpath:/home/zeroturnaround/jenkins-agent/workspace/javeleon-ejbtest-wildfly-latest/2f606c00/tmp/jrebel/lib/libjrebel64.so -Drebel.usage_reporting=false -Drebel.log=true -Drebel.log.file=/home/zeroturnaround/jenkins-agent/workspace/javeleon-ejbtest-wildfly-latest/2f606c00/log/wildfly31preview_suiteQaEjb3_packaged-modules_rebel.log -Drebel.log.perf_on_exit=true -Drebel.log.dump_stack_traces=60 -Drebel.loggers.trace=OpenWebBeans -Drebel.enable_test_hacks=true -Drebel.log.licensing=false -Drebel.mixpanel.url=http://127.0.0.1:44443 -Drebel.fatal_licensing=true -Drebel.preferred.license=1 -Drebel.license.email=EC2zt-devel-ubuntui-08e61c34226742374@zeroturnaround.com -Drebel.temp.dir=/home/zeroturnaround/jenkins-agent/workspace/javeleon-ejbtest-wildfly-latest/2f606c00/tmp/wildfly31preview -Drebel.remoting.cacheDir=/home/zeroturnaround/jenkins-agent/workspace/javeleon-ejbtest-wildfly-latest/2f606c00/tmp/wildfly31preview/remoting -Drebel.ide.console_log=false -Xlog:gc -Drebel.license.url=https://licenses-staging.zeroturnaround.com/e002d327-1a90-4b47-8034-d8aa078c9ed4 -XX:ReservedCodeCacheSize=96m -XX:MaxMetaspaceSize=1536m -Xmx1324m -Xss256k -Drebel.jboss7_deploy_hack=true -Drebel.detect_closed_zip=true --add-exports=java.desktop/sun.awt=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldaps=ALL-UNNAMED --add-exports=jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.management/javax.management=ALL-UNNAMED --add-opens=java.naming/javax.naming=ALL-UNNAMED -Djava.security.manager=allow -Dorg.jboss.boot.log.file=/opt/containers/wildfly-preview-31.0.1.Final/standalone/log/server.log -Dlogging.configuration=file:/opt/containers/wildfly-preview-31.0.1.Final/standalone/configuration/logging.properties 
java_command: /opt/containers/wildfly-preview-31.0.1.Final/jboss-modules.jar -mp /opt/containers/wildfly-preview-31.0.1.Final/modules org.jboss.as.standalone -Djboss.home.dir=/opt/containers/wildfly-preview-31.0.1.Final -Djboss.server.base.dir=/opt/containers/wildfly-preview-31.0.1.Final/standalone --server-config=standalone-full.xml
java_class_path (initial): /opt/containers/wildfly-preview-31.0.1.Final/jboss-modules.jar
Launcher Type: SUN_STANDARD

[Global flags]
     intx CICompilerCount                          = 4                                         {product} {ergonomic}
     uint ConcGCThreads                            = 2                                         {product} {ergonomic}
     uint G1ConcRefinementThreads                  = 8                                         {product} {ergonomic}
   size_t G1HeapRegionSize                         = 1048576                                   {product} {ergonomic}
    uintx GCDrainStackTargetSize                   = 64                                        {product} {ergonomic}
   size_t InitialHeapSize                          = 257949696                                 {product} {ergonomic}
   size_t MarkStackSize                            = 4194304                                   {product} {ergonomic}
   size_t MaxHeapSize                              = 1388314624                                {product} {command line}
   size_t MaxMetaspaceSize                         = 1610612736                                {product} {command line}
   size_t MaxNewSize                               = 832569344                                 {product} {ergonomic}
   size_t MinHeapDeltaBytes                        = 1048576                                   {product} {ergonomic}
   size_t MinHeapSize                              = 8388608                                   {product} {ergonomic}
    uintx NonNMethodCodeHeapSize                   = 4096                                   {pd product} {ergonomic}
    uintx NonProfiledCodeHeapSize                  = 0                                      {pd product} {ergonomic}
    uintx ProfiledCodeHeapSize                     = 0                                      {pd product} {ergonomic}
    uintx ReservedCodeCacheSize                    = 100663296                              {pd product} {command line}
   size_t SoftMaxHeapSize                          = 1388314624                             {manageable} {ergonomic}
     intx ThreadStackSize                          = 256                                    {pd product} {command line}
     bool UseCompressedOops                        = true                           {product lp64_product} {ergonomic}
     bool UseG1GC                                  = true                                      {product} {ergonomic}

Logging:
Log output configuration:
 #0: stdout all=warning,gc=info uptime,level,tags foldmultilines=false
 #1: stderr all=off uptime,level,tags foldmultilines=false

Environment Variables:
JAVA_HOME=/opt/java21-64
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
SHELL=/bin/bash
LANG=en_GB.UTF-8
LC_ALL=en_GB.UTF-8
TERM=unknown

Active Locale:
LC_ALL=en_GB.UTF-8
LC_COLLATE=en_GB.UTF-8
LC_CTYPE=en_GB.UTF-8
LC_MESSAGES=en_GB.UTF-8
LC_MONETARY=en_GB.UTF-8
LC_NUMERIC=en_GB.UTF-8
LC_TIME=en_GB.UTF-8

Signal Handlers:
   SIGSEGV: crash_handler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblocked
    SIGBUS: crash_handler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblocked
    SIGFPE: crash_handler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGPIPE: javaSignalHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, blocked
   SIGXFSZ: javaSignalHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, blocked
    SIGILL: crash_handler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblocked
   SIGUSR2: SR_handler in libjvm.so, mask=00000000000000000000000000000000, flags=SA_RESTART|SA_SIGINFO, blocked
    SIGHUP: UserHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, blocked
    SIGINT: SIG_IGN, mask=00000000000000000000000000000000, flags=none, blocked
   SIGTERM: UserHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, blocked
   SIGQUIT: UserHandler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, blocked
   SIGTRAP: crash_handler in libjvm.so, mask=11100100010111111101111111111110, flags=SA_RESTART|SA_SIGINFO, unblocked


Periodic native trim disabled

---------------  S Y S T E M  ---------------

OS:
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
uname: Linux 5.15.0-1055-aws #60~20.04.1-Ubuntu SMP Thu Feb 22 15:49:52 UTC 2024 x86_64
OS uptime: 0 days 2:14 hours
libc: glibc 2.31 NPTL 2.31 
rlimit (soft/hard): STACK 8192k/infinity , CORE 0k/infinity , NPROC 62476/62476 , NOFILE 81920/81920 , AS infinity/infinity , CPU infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity , MEMLOCK 65536k/65536k
load average: 22.45 20.21 20.85

/proc/meminfo:
MemTotal:       16009784 kB
MemFree:         2216672 kB
MemAvailable:    5652624 kB
Buffers:         1116300 kB
Cached:          1859636 kB
SwapCached:        11440 kB
Active:          2080556 kB
Inactive:       10585564 kB
Active(anon):     478160 kB
Inactive(anon):  9229048 kB
Active(file):    1602396 kB
Inactive(file):  1356516 kB
Unevictable:       25592 kB
Mlocked:           18476 kB
SwapTotal:       8388604 kB
SwapFree:        8144636 kB
Dirty:             18284 kB
Writeback:             0 kB
AnonPages:       9708720 kB
Mapped:           339728 kB
Shmem:               868 kB
KReclaimable:     813312 kB
Slab:             946152 kB
SReclaimable:     813312 kB
SUnreclaim:       132840 kB
KernelStack:       24256 kB
PageTables:        34432 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    16393496 kB
Committed_AS:   13731228 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       38612 kB
VmallocChunk:          0 kB
Percpu:             5920 kB
HardwareCorrupted:     0 kB
AnonHugePages:     28672 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      196516 kB
DirectMap2M:     8873984 kB
DirectMap1G:     7340032 kB

/sys/kernel/mm/transparent_hugepage/enabled: always [madvise] never
/sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter): always defer defer+madvise [madvise] never

Process Memory:
Virtual Size: 3318252K (peak: 3318252K)
Resident Set Size: 1698508K (peak: 1698508K) (anon: 1671228K, file: 27280K, shmem: 0K)
Swapped out: 0K
C-Heap outstanding allocations: 296301K, retained: 19606K
glibc malloc tunables: MALLOC_ARENA_MAX=1

/proc/sys/kernel/threads-max (system-wide limit on the number of threads): 124952
/proc/sys/vm/max_map_count (maximum number of memory map areas a process may have): 65530
/proc/sys/kernel/pid_max (system-wide limit on number of process identifiers): 4194304

container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-7
cpu_memory_nodes: 0
active_processor_count: 8
cpu_quota: no quota
cpu_period: 100000
cpu_shares: no shares
memory_limit_in_bytes: unlimited
memory_and_swap_limit_in_bytes: unlimited
memory_soft_limit_in_bytes: unlimited
memory_usage_in_bytes: 12414528 k
memory_max_usage_in_bytes: 14697736 k
kernel_memory_usage_in_bytes: 475288 k
kernel_memory_max_usage_in_bytes: unlimited
kernel_memory_limit_in_bytes: 1104800 k
maximum number of tasks: 18742
current number of tasks: 1233

KVM virtualization detected
Steal ticks since vm start: 2
Steal ticks percentage since vm start:  0.000

CPU: total 8 (initial active 8) (8 cores per cpu, 2 threads per core) family 25 model 1 stepping 1 microcode 0xa0011d1, cx8, cmov, fxsr, ht, mmx, 3dnowpref, sse, sse2, sse3, ssse3, sse4a, sse4.1, sse4.2, popcnt, lzcnt, tsc, tscinvbit, avx, avx2, aes, clmul, bmi1, bmi2, adx, sha, fma, vzeroupper, clflush, clflushopt, hv, rdtscp, rdpid, f16c
CPU Model and flags from /proc/cpuinfo:
model name	: AMD EPYC 7R13 Processor
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid

Online cpus: 0-7
Offline cpus: 
BIOS frequency limitation: <Not Available>
Frequency switch latency (ns): <Not Available>
Available cpu frequencies: <Not Available>
Current governor: <Not Available>
Core performance/turbo boost: <Not Available>

Memory: 4k page, physical 16009784k(5652624k free), swap 8388604k(8144636k free)
Page Sizes: 4k

vm_info: OpenJDK 64-Bit Server VM (21.0.2+13-LTS) for linux-amd64 JRE (21.0.2+13-LTS), built on 2024-01-16T00:00:00Z by "admin" with gcc 11.2.0

END.
@andresluuk andresluuk added the bug Something isn't working label Mar 4, 2024
@karianna
Copy link
Contributor

karianna commented Mar 5, 2024

@andresluuk Are you running the same workload (same versions etc) on Tomcat?

@andresluuk
Copy link
Author

Yes we are running the same workload (tests) on all the servers.
In more details we have multiple sets of tests and different suites have failed.
The memory overhead on Wildfly is probably more heavy then on tomcat, but this does not seem to be an oom issue as there are no extra active gc before the crashes. Because Wildfly has a complex deploy system probably it jits some parts more then tomcat, but from the crashes the latest jit list consist of different classes. Also the crash can happen relatively early on in the run, the dump I loaded here was ~1.5min after server start.
For reference I added another earlier crash to here.
hs_err_pid934085.log

@karianna karianna added jbs:needs-report Waiting for someone from our org to report to OpenJDK and removed Waiting on OP labels Mar 5, 2024
@FlorianKirmaier
Copy link

I have the same (similar?) crash.

See crash log.
hs_err_pid57242.log

V  [libjvm.dylib+0x8e9913]  OopStorage::release(oopDesc* const*)+0x53
V  [libjvm.dylib+0x3687ce]  CompileTask::free(CompileTask*)+0x8e
V  [libjvm.dylib+0x3612a6]  CompileQueue::get(CompilerThread*)+0x266
V  [libjvm.dylib+0x3648c2]  CompileBroker::compiler_thread_loop()+0x312
V  [libjvm.dylib+0x58ac79]  JavaThread::thread_main_inner()+0x99
V  [libjvm.dylib+0xb02e25]  Thread::call_run()+0xa5
V  [libjvm.dylib+0x8f076e]  thread_native_entry(Thread*)+0x10e

Stack looks slightly different.

@karianna
Copy link
Contributor

karianna commented Jun 1, 2024

I have the same (similar?) crash.

See crash log. hs_err_pid57242.log

V  [libjvm.dylib+0x8e9913]  OopStorage::release(oopDesc* const*)+0x53
V  [libjvm.dylib+0x3687ce]  CompileTask::free(CompileTask*)+0x8e
V  [libjvm.dylib+0x3612a6]  CompileQueue::get(CompilerThread*)+0x266
V  [libjvm.dylib+0x3648c2]  CompileBroker::compiler_thread_loop()+0x312
V  [libjvm.dylib+0x58ac79]  JavaThread::thread_main_inner()+0x99
V  [libjvm.dylib+0xb02e25]  Thread::call_run()+0xa5
V  [libjvm.dylib+0x8f076e]  thread_native_entry(Thread*)+0x10e

Stack looks slightly different.

Could be, if you can replicate with 21.0.3 that would be ace.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working jbs:needs-report Waiting for someone from our org to report to OpenJDK
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@karianna @andresluuk @FlorianKirmaier