Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Java crash on M4 Macs causing OCIS docker container restart loop #10965

Closed
prohtex opened this issue Feb 5, 2025 · 11 comments
Closed

Java crash on M4 Macs causing OCIS docker container restart loop #10965

prohtex opened this issue Feb 5, 2025 · 11 comments

Comments

@prohtex
Copy link

prohtex commented Feb 5, 2025

I have a new WOPI deployment running in a Ubuntu 24.04.1 VM on an M4 Mac host. After finishing the deployment, the Tika container keeps restarting with a Java crash.

# docker container list
CONTAINER ID   IMAGE                          COMMAND                  CREATED          STATUS                            PORTS                                                                      NAMES
375646d0829c   owncloud/ocis-rolling:latest   "/bin/sh -c 'ocis co…"   13 minutes ago   Up 3 minutes                      9200/tcp                                                                   ocis_full-collaboration-1
58f6ff00938d   owncloud/ocis-rolling:latest   "/bin/sh -c 'ocis in…"   13 minutes ago   Up 10 seconds                     9200/tcp                                                                   ocis_full-ocis-1
6898ae39998a   traefik:v3.2.2                 "/entrypoint.sh --lo…"   13 minutes ago   Up 13 minutes                     0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   ocis_full-traefik-1
6d7fcc2442e1   collabora/code:24.04.10.2.1    "bash -c 'coolconfig…"   13 minutes ago   Up 13 minutes (healthy)           9980/tcp                                                                   ocis_full-collabora-1
3eaa53f68e32   apache/tika:latest-full        "/bin/sh -c 'exec ja…"   13 minutes ago   Restarting (127) 32 seconds ago         

The ocis-rolling container and apache containers last a matter of seconds before restarting.

The logs look like this:

$ sudo docker logs --tail 50 --follow --timestamps 3eaa53f68e32
2025-02-05T04:50:12.047498970Z # /tmp/hs_err_pid1.log
2025-02-05T04:50:12.048156460Z [0.010s][warning][os] Loading hsdis library failed
2025-02-05T04:50:12.050759802Z #
2025-02-05T04:50:12.050772424Z # The crash happened outside the Java Virtual Machine in native code.
2025-02-05T04:50:12.050774132Z # See problematic frame for where to report the bug.
2025-02-05T04:50:12.050787921Z #
2025-02-05T04:50:12.050840285Z 
2025-02-05T04:50:12.050863697Z [error occurred during error reporting (), id 0x5, SIGTRAP (0x5) at pc=0x0000fbc1be941b38]
2025-02-05T04:51:12.147923549Z #
2025-02-05T04:51:12.147935632Z # A fatal error has been detected by the Java Runtime Environment:
2025-02-05T04:51:12.147936757Z #
2025-02-05T04:51:12.147937465Z #  SIGILL (0x4) at pc=0x0000ee7f37d40c5c, pid=1, tid=7
2025-02-05T04:51:12.147938382Z #
2025-02-05T04:51:12.147939090Z # JRE version:  (21.0.5+11) (build )
2025-02-05T04:51:12.147939923Z # Java VM: OpenJDK 64-Bit Server VM (21.0.5+11-Ubuntu-1ubuntu124.10, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
2025-02-05T04:51:12.147945631Z # Problematic frame:
2025-02-05T04:51:12.147946590Z # j  java.lang.System.registerNatives()V+0 [email protected]
2025-02-05T04:51:12.147947048Z #
2025-02-05T04:51:12.147947506Z # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to //core.1)
2025-02-05T04:51:12.147948090Z #
2025-02-05T04:51:12.147950756Z # An error report file with more information is saved as:
2025-02-05T04:51:12.147965714Z # /tmp/hs_err_pid1.log
2025-02-05T04:51:12.150573964Z [0.010s][warning][os] Loading hsdis library failed
2025-02-05T04:51:12.153132590Z #
2025-02-05T04:51:12.153135423Z # The crash happened outside the Java Virtual Machine in native code.
2025-02-05T04:51:12.153136548Z # See problematic frame for where to report the bug.
2025-02-05T04:51:12.153137340Z #
2025-02-05T04:51:12.153173922Z 
2025-02-05T04:51:12.153176172Z [error occurred during error reporting (), id 0x5, SIGTRAP (0x5) at pc=0x0000ee7f506f1b38]
2025-02-05T04:52:12.254493389Z #
2025-02-05T04:52:12.254553134Z # A fatal error has been detected by the Java Runtime Environment:
2025-02-05T04:52:12.254557508Z #
2025-02-05T04:52:12.254558591Z #  SIGILL (0x4) at pc=0x0000ed7c9fd40c5c, pid=1, tid=7
2025-02-05T04:52:12.254561966Z #
2025-02-05T04:52:12.254574673Z # JRE version:  (21.0.5+11) (build )
2025-02-05T04:52:12.254591464Z # Java VM: OpenJDK 64-Bit Server VM (21.0.5+11-Ubuntu-1ubuntu124.10, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
2025-02-05T04:52:12.254593672Z # Problematic frame:
2025-02-05T04:52:12.254644626Z # j  java.lang.System.registerNatives()V+0 [email protected]
2025-02-05T04:52:12.254665874Z #
2025-02-05T04:52:12.254667457Z # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to //core.1)
2025-02-05T04:52:12.254668582Z #
2025-02-05T04:52:12.254728702Z # An error report file with more information is saved as:
2025-02-05T04:52:12.254743909Z # /tmp/hs_err_pid1.log
2025-02-05T04:52:12.257321274Z [0.010s][warning][os] Loading hsdis library failed
2025-02-05T04:52:12.259905263Z #
2025-02-05T04:52:12.259914429Z # The crash happened outside the Java Virtual Machine in native code.
2025-02-05T04:52:12.259915679Z # See problematic frame for where to report the bug.
2025-02-05T04:52:12.259944010Z #
2025-02-05T04:52:12.260000297Z 
2025-02-05T04:52:12.260029878Z [error occurred during error reporting (), id 0x5, SIGTRAP (0x5) at pc=0x0000ed7cb81c1b38]

When I migrate this VM to an M1 CPU host with otherwise identical configuration, it works fine:

# docker container list
CONTAINER ID   IMAGE                          COMMAND                  CREATED        STATUS                  PORTS                                                                      NAMES
de5069340ed6   owncloud/ocis-rolling:latest   "/bin/sh -c 'ocis co…"   45 hours ago   Up 29 hours             9200/tcp                                                                   ocis_full-collaboration-1
e1f68dfb7e85   owncloud/ocis-rolling:latest   "/bin/sh -c 'ocis in…"   45 hours ago   Up 29 hours             9200/tcp                                                                   ocis_full-ocis-1
c518a5d51e74   apache/tika:latest-full        "/bin/sh -c 'exec ja…"   45 hours ago   Up 29 hours             9998/tcp                                                                   ocis_full-tika-1
9582b0d44329   collabora/code:24.04.10.2.1    "bash -c 'coolconfig…"   45 hours ago   Up 29 hours (healthy)   9980/tcp                                                                   ocis_full-collabora-1
2224657e4b19   traefik:v3.2.2                 "/entrypoint.sh --lo…"   45 hours ago   Up 29 hours             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   ocis_full-traefik-1

Any ideas how to rectify?

@prohtex prohtex changed the title OCIS docker containers stuck in restart loop Java crash on M4 Macs causing OCIS docker container restart loop Feb 5, 2025
@prohtex
Copy link
Author

prohtex commented Feb 5, 2025

This seems to be an M4-specific issue:

corretto/corretto-21#85

https://bugs.openjdk.org/browse/JDK-8345296

@prohtex
Copy link
Author

prohtex commented Feb 5, 2025

Per this link, the fix seems to be JAVA_OPTS=“-XX:UseSVE=0” but I'm not sure where to put that...

https://forums.docker.com/t/image-builds-fail-on-new-macbook-despite-working-fine-on-prior-apple-silicon/145772/5

@micbar @ScharfViktor please help!

@prohtex prohtex mentioned this issue Feb 5, 2025
40 tasks
@saw-jan
Copy link
Member

saw-jan commented Feb 5, 2025

Per this link, the fix seems to be JAVA_OPTS=“-XX:UseSVE=0” but I'm not sure where to put that...

This might be in the tika service:

tika:
image: ${TIKA_IMAGE:-apache/tika:latest-full}
# release notes: https://tika.apache.org
networks:
ocis-net:
restart: always
logging:
driver: ${LOG_DRIVER:-local}

You can check if this helps:

  tika:
    image: ${TIKA_IMAGE:-apache/tika:latest-full}
    # release notes: https://tika.apache.org
    networks:
      ocis-net:
    restart: always
    environment:
      JAVA_OPTS: "-XX:UseSVE=0"
    logging:
      driver: ${LOG_DRIVER:-local}

@saw-jan
Copy link
Member

saw-jan commented Feb 5, 2025

CC @kobergj @2403905

@prohtex
Copy link
Author

prohtex commented Feb 5, 2025

tika:
image: ${TIKA_IMAGE:-apache/tika:latest-full}
# release notes: https://tika.apache.org
networks:
ocis-net:
restart: always
environment:
JAVA_OPTS: "-XX:UseSVE=0"
logging:
driver: ${LOG_DRIVER:-local}

Hi @saw-jan! Tried that, but no luck so far. Also tried adding to .env. I thought maybe something needed to be compiled in, like an ldflag?

@wkloucek
Copy link
Contributor

wkloucek commented Feb 5, 2025

oCIS does not use JAVA.

3eaa53f68e32 apache/tika:latest-full "/bin/sh -c 'exec ja…" 13 minutes ago Restarting (127) 32 seconds ago

You can see that it's Tika. This is not an oCIS bug.

I think it would be a better fit for the Tika issue tracker: https://issues.apache.org/jira/projects/TIKA/issues/

@wkloucek
Copy link
Contributor

wkloucek commented Feb 5, 2025

Per this link, the fix seems to be JAVA_OPTS=“-XX:UseSVE=0” but I'm not sure where to put that...

see also eg. https://stackoverflow.com/questions/5241743/how-do-i-use-the-java-opts-environment-variable

This means it needs to go in the docker compose environment section of the tika service, see also https://docs.docker.com/compose/how-tos/environment-variables/set-environment-variables/

@prohtex
Copy link
Author

prohtex commented Feb 5, 2025

Per this link, the fix seems to be JAVA_OPTS=“-XX:UseSVE=0” but I'm not sure where to put that...

see also eg. https://stackoverflow.com/questions/5241743/how-do-i-use-the-java-opts-environment-variable

This means it needs to go in the docker compose environment section of the tika service, see also https://docs.docker.com/compose/how-tos/environment-variables/set-environment-variables/

Thank you @wkloucek, I will log the issue with the Tika team. I added the env var per @saw-jan suggestion, but it didn't work. Is there somewhere else in the container it needs to be set?

I tried:

-adding to .env as JAVA_OPTS="-XX:UseSVE=0"
-adding to tika.yml as JAVA_OPTS: -XX:UseSVE=0
-adding to tika.yml as JAVA_OPTS: "-XX:UseSVE=0"

@saw-jan
Copy link
Member

saw-jan commented Feb 5, 2025

Maybe try JAVA_TOOL_OPTIONS: "-XX:UseSVE=0"?
Suggested in https://stackoverflow.com/questions/5241743/how-do-i-use-the-java-opts-environment-variable

@prohtex
Copy link
Author

prohtex commented Feb 5, 2025

Maybe try JAVA_TOOL_OPTIONS: "-XX:UseSVE=0"? Suggested in https://stackoverflow.com/questions/5241743/how-do-i-use-the-java-opts-environment-variable

@saw-jan you are my hero! That did it :)

@prohtex
Copy link
Author

prohtex commented Feb 5, 2025

@wkloucek I will log my issue with Tika but in the meantime, it seems adding this to the ocis_full deployment example would be a good idea. Thanks everyone.

@prohtex prohtex closed this as completed Feb 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants