Running SwarmSpawner with c.SwarmSpawner.extra_create_kwargs = {"user": "root"} fails

[phockett]

Bug description

I have SwarmSpawner working on a cluster (details below), but would like to run JupyterStacks containers with the logged in username and UID. Running a container directly as per the docs works correctly, e.g. something like:

  docker run -it --rm \
      -p 803:8888 \
      -e NB_USER="testUser" \
      -e CHOWN_HOME=yes \
      -e NB_UID=1045 \
      -e NB_GID=100 \
      jupyter/datascience-notebook

With SwarmSpawner I can pass the env settings without issues (see note below on the method), but my spawn fails when I add c.SwarmSpawner.extra_create_kwargs = {"user": "root"} (as specified for changing the user in the SwarmSpawner docs)

Note: for user management I'm using LDAP, and pulling the user info with a pre-spawn-hook routine (see config section below). The working environmental settings go via lines like spawner.environment['NB_UID'] = str(spawner.userdata['uidNumber'][0]), where userdata is populated for the given user from the LDAP server.

In my case I'm stuck running a fairly old version of Docker (20.10.7, API 1.41), so I'm guessing this is the cause of the issue, but it would be good to confirm this, and not just that I'm doing something stupid!

Work-around: building images from Docker Stacks locally and setting the user to root avoids the need to pass --user root, and works as desired with the SwarmSpawner and passed env settings in my case. A minimal Dockerfile example for this is simply:

ARG JUPYTERSTACKS_TAG="2023-09-11"
ARG JUPYTERSTACKS_NOTEBOOK="datascience-notebook"

FROM jupyter/${JUPYTERSTACKS_NOTEBOOK}:${JUPYTERSTACKS_TAG}

USER root

In this case, containers spawn as expected, and accept the additional user environmental settings passed at spawn.

How to reproduce

Add the line c.SwarmSpawner.extra_create_kwargs = {"user": "root"} to a working SwarmSpawner jupyterhub_config.py and test!

Expected behaviour

Equivalent to docker run --user root as per the documentation.

Actual behaviour

Error on Spawn, specifically TypeError: ServiceApiMixin.create_service() got an unexpected keyword argument 'user'. (I also tested for "--user", "u" and "-u" instead of "user", but got the same result.)

jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    | [E 2023-09-14 22:32:49.530 JupyterHub user:884] Unhandled error starting testUser's server: ServiceApiMixin.create_service() got an unexpected keyword argument 'user'
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |     Traceback (most recent call last):
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 798, in spawn
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |         url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1270, in start
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |         obj = await self.create_object()
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/swarmspawner.py", line 239, in create_object
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |         service = await self.docker("create_service", **create_kwargs)
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |       File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |         result = self.fn(*self.args, **self.kwargs)
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 948, in _docker
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |         return m(*args, **kwargs)
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |       File "/usr/local/lib/python3.10/dist-packages/docker/utils/decorators.py", line 34, in wrapper
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |         return f(self, *args, **kwargs)
jupyterhub-SERVER_jupyterhub.1.kawjv7v5x4hs@SERVER    |     TypeError: ServiceApiMixin.create_service() got an unexpected keyword argument 'user'

Your personal set up

• OS: cat /etc/lsb-release

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

• Docker version:

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:47 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

• Jupyter Hub container build from v4.0.2 Docker Hub image (Python 3.10), with current versions of LDAP authenticator (v1.3.2) and Docker Spawner (v12.1.0) pulled from Pypi:

ARG JUPYTERHUB_VERSION=4.0.2 
FROM jupyterhub/jupyterhub:${JUPYTERHUB_VERSION}

RUN pip install \
    dockerspawner \
    jupyterhub-ldapauthenticator
    
# For LDAP testing
RUN apt-get update && apt install -y slapd ldap-utils \
    && apt install -y iproute2 iputils-ping

Configuration

# [...Messy test config - SwarmSpawner settings only below...]

# **** SWARM Spawner settings
# Based on example from https://zozoo.io/how-to-run-jupyterhub-in-docker-swarm-environment-using-swarmspawner/
# NOT sure all settings are required, but this config worked in testing.

c.ConfigurableHTTPProxy.should_start = True
c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'
c.SwarmSpawner.network_name = network_name
c.SwarmSpawner.extra_host_config = { 'network_mode': network_name }
#c.SwarmSpawner.extra_create_kwargs.update({ 'volume_driver': 'local' })
c.SwarmSpawner.remove_containers = True
c.SwarmSpawner.debug = True
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.hub_port = jhub_port
c.SwarmSpawner.host_ip = "0.0.0.0"

# TLS config
c.JupyterHub.port = 8000
#c.JupyterHub.ssl_key = os.environ['SSL_KEY']
#c.JupyterHub.ssl_cert = os.environ['SSL_CERT']

c.SwarmSpawner.http_timeout = 300
c.SwarmSpawner.start_timeout = 3000   # Set longer timeout here to allow for disemination of containers to nodes

c.SwarmSpawner.extra_placement_spec = { 'constraints' : ['node.role==worker'] }

# GET additional user info via LDAP
# See https://github.com/jupyterhub/kubespawner/issues/298#issuecomment-466440455
# Note auth_state from ldapauth should have this, but not working in testing, so use custom routine.
def pre_spawn_start_lookup(spawner):  
    user = spawner.user.name
    
    # Try custom lookup, LDAPauth doesn't seem to be doing as advertised? (No additional data pulled, and nothing in logs)
    from ldap3 import Server, Connection, ALL
    server = Server(serverName,  get_info=ALL)
    conn = Connection(server, auto_bind=True)
    conn.search(f'dc={serverName},dc=local', f'(&(objectclass=person)(uid={user}))',attributes=['cn', 'givenName', 
                'uidNumber','homeDirectory','gidNumber'])
    
    spawner.userdata = conn.entries[0]
    spawner.log.info(f"Got {user} additional props.")
    spawner.log.info(spawner.userdata)
    
    # setup environment
    spawner.environment = {
        "CHOWN_HOME": "yes",
        "CHOWN_EXTRA": "/home/jovyan",
        "CHOWN_HOME_OPTS": "-R",
    }
    
    # Works if JupyterLab runs as root user
    spawner.environment['NB_UID'] = str(spawner.userdata['uidNumber'][0])
    spawner.environment['NB_USER'] = user
	

c.SwarmSpawner.debug = True
c.Spawner.pre_spawn_hook = pre_spawn_start_lookup

# Specify root user? THIS FAILS
# c.SwarmSpawner.extra_create_kwargs = {"user": "root"}

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉