Error on loading: Failed to install package when not running as root

[kbirger]

What version of Music Assistant has the issue?

2.2.5

What version of the Home Assistant Integration have you got installed?

2024.9.1

Have you tried everything in the Troubleshooting FAQ and reviewed the Open and Closed Issues and Discussions to resolve this yourself?

• Yes

The problem

Upon load, I see the following error in logs
2024-09-23 12:29:54.283 INFO (MainThread) [music_assistant] Starting Music Assistant Server (865a11d3ada243cebbe04607c469bf53) version 2.2.5 - HA add-on: False - Safe mode: False
2024-09-23 12:29:54.288 INFO (MainThread) [music_assistant.cache] Initializing cache controller...
2024-09-23 12:29:54.362 INFO (MainThread) [music_assistant.music] Using a sync interval of 180 minutes.
2024-09-23 12:29:54.401 INFO (MainThread) [music_assistant.streams] Detected ffmpeg version 6.1.1 with libsoxr support
2024-09-23 12:29:54.401 INFO (MainThread) [music_assistant.streams] Starting server on 0.0.0.0:8097 - base url: http://192.168.1.111:8097
2024-09-23 12:29:54.406 INFO (MainThread) [music_assistant.webserver] Starting server on 0.0.0.0:8095 - base url: http://192.168.1.111:8095
2024-09-23 12:29:54.898 ERROR (MainThread) [music_assistant] Error doing task: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/aiorun.py", line 219, in new_coro
await coro
File "/usr/local/lib/python3.12/site-packages/music_assistant/main.py", line 212, in start_mass
await mass.start()
File "/usr/local/lib/python3.12/site-packages/music_assistant/server/server.py", line 161, in start
await self._load_providers()
File "/usr/local/lib/python3.12/site-packages/music_assistant/server/server.py", line 549, in _load_providers
prov_configs = await self.config.get_provider_configs(include_values=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/music_assistant/server/controllers/config.py", line 169, in get_provider_configs
await self.get_provider_config(prov_conf["instance_id"])
File "/usr/local/lib/python3.12/site-packages/music_assistant/server/controllers/config.py", line 183, in get_provider_config
config_entries = await self.get_provider_config_entries(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/music_assistant/server/controllers/config.py", line 233, in get_provider_config_entries
prov_mod = await load_provider_module(provider_domain, prov.requirements)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/music_assistant/server/helpers/util.py", line 116, in load_provider_module
await install_package(requirement)
File "/usr/local/lib/python3.12/site-packages/music_assistant/server/helpers/util.py", line 49, in install_package
raise RuntimeError(msg)
RuntimeError: Failed to install package aiojellyfin==0.10.1
error: failed to create file /tmp/uv-39ee7c309557c5e9.lock
Caused by: Permission denied (os error 13)

I get a similar error with hass-client.

It seems like it wants to install packages, but has an RO /data mount.

How to reproduce

Load up app

Music Providers

none

Player Providers

Home assistant

Full log output

log.txt

Additional information

It seems like the latest version of the server package requires you to run as root in order to install some dependencies. In my opinion, that's not a good reason. This was not an issue in previous versions. Why is it no longer possible to install these dependencies during build of the image? The second best solution would be to set up the container in such a way that the dependencies are installed to a location that the container user should have write access to. Third best would be to support environment variables like the linuxserver images that would drop root after the dependencies are installed.

What version of Home Assistant Core are your running

2024.9.1

What type of installation are you running?

Home Assistant Container

On what type of hardware are you running?

Linux

[marcelveldt]

You need to run it in privileged mode (as written in the documentation) until someone creates an apparmor profile

[kbirger]

Thank you for your reply. I'm already running in privileged mode, although I don't understand why that should be required just to update some Python packages inside the container's internal volume.

Below is a snippet of my compose. I have commented the user lines to make it work, but previously it was working without user and without the extra capabilities.

music_assistant:
    container_name: music_assistant
    image: ghcr.io/music-assistant/server
    # user: ${MASS_PUID}:${MASS_PGID}
    privileged: true
    network_mode: host
    restart: unless-stopped
    cap_add:
      - SYS_ADMIN
      - DAC_READ_SEARCH
    ports:
    - 8095:8095
    - 8097:8097
    volumes:
    - /docker/data/music-assistant:/data:rw
    - /share/music:/media:ro

It looks like when the code goes to install the package, it tries to create a lock, but this lock file already exists, and was created by root:

-rw-r--r--    1 root     root             0 Sep 16 22:46 uv-39ee7c309557c5e9.lock

[kbirger]

It looks like that lock file is getting created during the build and not getting cleaned up. It's present before the mass process even starts. I tested by running

docker run --rm -it --name matest --privileged -v /docker/data/music-assistant:/data:rw --user=XXX:XXX --entrypoint sh ghcr.io/music-assistant/server

It's probably being created by the uv pip install in the base image

[marcelveldt]

I suspect this is an issue with uv (which we use to speedup package installs) - I will add a quick fallback to regular pip

[marcelveldt]

Can you check if this is fixed in the beta ? 2.3.0 beta 27 is what has the patch to fallback to regular pip.
If you dont want to try the beta, you'll have to wait a bit until we release a new stable(path) release.

[kbirger]

I don't mind beta software. I gave it a shot. The install error is gone, but I get a different error.

sqlite3.OperationalError: no such column: key

Migration didn't run maybe?
error2.txt

Why do you need to fall back to UV? shouldn't it be fairly straight forward to have the build step clean up its lock file?

[OzGav]

Were you upgrading a beta install?

[kbirger]

No, I'm coming from the latest stable. Is there a process for going to a beta?

I would like to confirm that the issue did indeed arise in version 2.2.3, and also that it seems insufficient to simply delete the lock file, as I initially suggested. Somehow PIP is fine with the file system being owned by root and restricted, but UV is not.

One other option, which is pip's own recommendation, is to use a virtual env.
..or just to chmod the filesystem to accomodate for non-root users

[I-G-1-1]

I have a similar issue. The last version working with my configuration is 2.3.0b8.
I saw 2.3.0b27 changelog with "Fallback to regular pip in case uv pip fails due to permissions" and give it a try.
The package install fails with this error:

Installing collected packages: urllib3, protobuf, charset-normalizer, requests, casttube, PyChromecast
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/.local'
Check the permissions.

Here is my docker compose file (note that I also tried to run the docker image with privileged: true):

services:
  music-assistant-server:
    image: ghcr.io/music-assistant/server:beta
    container_name: music-assistant-server
    user: 1001:1001 # musicassistant should be owner of volumes
    ports:
      - "8095:8095"
      - "8098:8098"
    restart: unless-stopped
    # Network mode must be set to host for MA to work correctly
    network_mode: host
    volumes:
      - /home/musicassistant/.config/musicassistant/data:/data/
    # privileged caps needed to mount smb folders within the container
    #cap_add:
      #- SYS_ADMIN
      #- DAC_READ_SEARCH
    #privileged: true
    privileged: false
    environment:
      # Provide logging level as environment variable.
      # default=info, possible=(critical, error, warning, info, debug, verbose)
      - LOG_LEVEL=info
      #- LOG_LEVEL=debug
      - PUID=1001
      - PGID=1001

Here are the log files:
unprivileged:
music-assistant-v2.3.0b27-unprivileged.log
privileged:
music-assistant-v2.3.0b27-privileged.log

[OzGav]

@I-G-1-1 for support you need to run the container in privileged mode. I know you said you did but you need to post any logs or compose files with that set to true

[rwlove]

Sane problem here, when running 2.3.0b27 as a container in a K8S cluster:

ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/.local'

[kbirger]

Some information about privileged mode:
https://docs.docker.com/engine/containers/run/#runtime-privilege-and-linux-capabilities

https://learn.snyk.io/lesson/container-runs-in-privileged-mode/

The documentation for music assistant regarding this subject is here.
https://music-assistant.github.io/installation/

It's a little ambiguous, but it sounds like officially privileged mode is required for accessing shares from within the container. It makes sense that that would require privileged mode, but I think it's a bad idea to do that anyway. In another issue, @I-G-1-1 notes that privileged mode is required as of new betas for streaming.

In the sample docker compose file, privileged is set along with some explicit cap_add statements. I'm not an expert here, but isn't that actually redundant?
https://github.com/music-assistant/server/blob/main/docker-compose.example.yml

After all of this, it is still unclear why we are taking about privileged mode here in this issue. Again, I'm not an expert, so I welcome someone to educate me, but my understanding is that privileged mode is not about file system permissions within the container, but about elevated privileges to access certain apis and devices on the host.

This matches with what I'm seeing in terms of running with privileged mode, but not having any success unless I also run the container as root, which is doubly bad, because that means that this container is essentially running as root on my host.

[marcelveldt]

After all of this, it is still unclear why we are taking about privileged mode here in this issue. Again, I'm not an expert, so I welcome someone to educate me, but my understanding is that privileged mode is not about file system permissions within the container, but about elevated privileges to access certain apis and devices on the host.

We require privileged mode for the SMB provider, otherwise mounting of samba shares fails. It could be that it also caused other trouble in the past without it enabled but I'm not sure. Most important is the host networking, so that the multicast mdns/upno traffic works out of the box to discover players.

It will probably also work without privileged and a more finegrained access rights control but so far nobody with more knowledge about this stood up to fix that.

We primary target on the Home Assistant add-on (as that is what most people use) and the docker image is just an additional install method but we simply are so underpowered that we do not spend a lot of time tweaking it. I hope somebody reads this and can help us fixing this properly.

Until then, I'll see if I can figure out why the pip install commands are failing on docker and not on our HA add-on.

[marcelveldt]

@kbirger I missed some of your comments so here a few replies:

Why is it no longer possible to install these dependencies during build of the image?

That is done just to slim down the size of the image as we are expanding with more and more providers (and thus optional packages) so we only include the bare minimum in the base image and install the rest on the fly. Now that we switched to the alpine image, we can use the prebuilt wheels from HA.

It looks like that lock file is getting created during the build and not getting cleaned up. It's probably being created by the uv pip install in the base image

Ah, yes that must be it. I'll look in that direction. In the meanwhile I've spinned up a plain Debian install to test the docker image.

[marcelveldt]

Yes, indeed that tempfile is a leftover from the base image creation.
Also figured out why we dont run into this issue on the addon; there we map a memorydrive to /tmp

[kbirger]

Thanks for looking into this. I'm happy to test out anything that would help

[kbirger]

I just created a PR music-assistant/server#1682. Please look when you have a chance. I wasn't able to test fully because I'm not too familiar with wheels or muslinux, and I could not get it to pull down the package during my release step.

The concept works though. If I pull the updated base image and run a shell on it, the shell is not root, and I'm able to run an install for some public package using uv pip install.

One thing that might still be wrong is, I'm not sure if sourcing the activate script for the venv carries over into the child image. That step might need to be moved.

If you can advise how I should go about testing this end to end on a fork repo, that would be welcome. You're also welcome to take these small changes and fix them up yourself.

The general idea is that you create a specific user in the base image, complete all of your root tasks, such as assigning directory access, then drop root by switching to that user.

I thought that using a virtual environment would eliminate the need to do extra steps, but it seems like on alpine linux, everything is only writable by the root user. It's probably possible to skip doing a virtual env altogether, but it never hurts IMO.

Thanks for your work.

[marcelveldt]

Hi, I had already been working on this yesterday and did not report back here.

I had already made some changes to the base image to allow it to run as another user yourself, just by fixing the permissions on the venv and temp. That way it would would for both the HA addon and your situation running it as another user.

I did also test your approach but it caused other issues again with cifs mounting for example.

I can test it one more time (because you have a slightly different order than my tests) but I'm afraid it wont work.

[jojo141185]

@marcelveldt Is it possible to test an image with your customizations?
I just encountered this problem during the initial installation of musicassistant while setting up new music providers. I want to start the container under a custom user with limited permissions instead of root.

[marcelveldt]

@marcelveldt Is it possible to test an image with your customizations? I just encountered this problem during the initial installation of musicassistant while setting up new music providers. I want to start the container under a custom user with limited permissions instead of root.

Do you want to be able to run the SMB provider or will you mount your media files yourself (or media not use local files at all) ?

[jojo141185]

I don't use CIFS/SMB mount. I add my local media files via the Docker volume.

[kbirger]

@marcelveldt thank you for taking a look. I'll update according to your recommendation.

Honestly though, I would just document that smb through docker is not officially supported. Both @jojo141185 and I use SMB, and we are both doing it through the host filesystem. I'm not arrogant enough to say that we are the only ones that matter, but I do think that this is the correct approach. It is more secure, more reliable, and simpler. There is no need to have the container do this. When running in addon mode, you have no option but to support this, and it's wonderful that MASS has the ability to do it, but for docker use cases, it's just not the best way.

I'm happy to share a guide for how to properly set up an SMB share on Ubuntu server.

[marcelveldt]

Yeah, I 100% agree with you. The only thing I need to double check if this doesn't break the HA add-on (because that uses default root + apparmor) - I will double check that and then we're all set.

And yes, I think its fair to just say that mounting samba within the docker container needs either the root workaround or just mount it at host level (the better way)

[marcelveldt]

OK, it completely breaks the HA addons so I've (partially) reverted it, sorry.
Instead the container now runs at default as root but you can simply start it with your own user instead, so e.g.:

docker run -v <dir>:/data --network host --user 1001:1001 ghcr.io/music-assistant/server:2.3.0b28

[OzGav]

@kbirger Beta28 is released

[marcelveldt]

Did one of you test 2.3.0b28 already ? I'm curious if everything works now.
If it works, we'll adjust the documentation (about running as a different user) and also release a new stable patch.

[I-G-1-1]

I'm running v2.3.0b28 and it seems to work fine in privileged mode.

As I don't use "remote share" as music provider (in case I'll mount my network drives on the host) I also tried the unprivileged mode and it seems to work fine.

here is my docker compose file if it could help for the documentation:

services:
  music-assistant-server:
    image: ghcr.io/music-assistant/server:beta # <<< Desired release version here (change beta to latest once there is a stable release)
    container_name: music-assistant-server
    user: 1001:1001 # musicassistant should be owner of volumes
    ports:
      - "8095:8095"
      - "8098:8098"
    restart: unless-stopped
    # Network mode must be set to host for MA to work correctly
    network_mode: host
    volumes:
      #- ${USERDIR:-$HOME}/docker/music-assistant-server/data:/data/
      - /home/musicassistant/.config/musicassistant/data:/data/
    # privileged caps needed to mount smb folders within the container
    #cap_add:
      #- SYS_ADMIN
      #- DAC_READ_SEARCH
    #privileged: true
    privileged: false
    environment:
      # Provide logging level as environment variable.
      # default=info, possible=(critical, error, warning, info, debug, verbose)
      - LOG_LEVEL=info
      #- LOG_LEVEL=debug
      - PUID=1001
      - PGID=1001

@marcelveldt thanks for your work for fix this issue, really appreciated!

Here are the log files of v2.3.0b28:
privileged:
music-assistant-v2.3.0b28-privileged.log
unprivileged:
music-assistant-v2.3.0b28-unprivileged.log

[marcelveldt]

@I-G-1-1 you don't need privileged at all. You only need the "sys_admin" and "dac_read_search" if you want to mount filesystems (but that also requires root user).

You can drop the PUID and PGID env vars from the docker compose, we don't use them.

Please also test playback. Look out for this error being logged:

"Detected that you are running the (docker) container without permissive access rights. This will impact performance !"

If that is the case, you might need to add the SYS_ADMIN cap again. We are increasing some pipe sizes to work with larger buffers (to smoothen audio streaming) but that failed on previous python versions.

If playback works just fine and no error is being logged, we can maybe remove that check from the code now.

[I-G-1-1]

I can playback music.

The error you refer to it's not logged, but I have LOG_LEVEL=info, do I need to put LOG_LEVEL=debug to see that error?

[kbirger]

@marcelveldt 2.3.0b28 is working perfectly for me. Thanks so much.

For reference, in case this is helpful to you @I-G-1-1, see my docker-compose config

music_assistant:
    container_name: music_assistant
    image: ghcr.io/music-assistant/server:2.3.0b28
    user: ${MASS_PUID}:${MASS_PGID}
    privileged: true
    network_mode: host
    restart: unless-stopped
    cap_add:
      - SYS_ADMIN
      - DAC_READ_SEARCH
    ports:
    - 8095:8095
    - 8097:8097
    volumes:
    - /docker/data/music-assistant:/data:rw
    - /share/music:/media:ro

Marcel, I didn't realize that the addon uses docker at all. I'm curious - what can I search for to learn more?

[marcelveldt]

Marcel, I didn't realize that the addon uses docker at all. I'm curious - what can I search for to learn more?

Yes Home Assistant add-ons are just docker containers under the hood but with an additional management and security layer on top. Have a look at developer.home-assistant.io (section about the supervisor and add-ons) for more info.

[jojo141185]

I tested the docker image of the new beta version today. Everything seems to work now, even if the container is started under a different user (not root).
Thank you Marcel for your support!