Spawn notebook servers using host network in Docker swarm using swarmspawner.

[sangramga]

I need to spawn notebook server in Docker Swarm, and these containers must be in host network (instead of overlay network).

I am running Docker swarm with 2 nodes (one master and one worker). I run jupyterhub as docker service on master node as specified in Readme.I am using dockerspawner from Github master repo. Also I am using jupyterhub-0.9
My Dockerfile for jupyterhub:

FROM jupyterhub/jupyterhub:0.9

EXPOSE 8081

RUN /opt/conda/bin/pip install jupyterhub-hashauthenticator jupyter_client
COPY dockerspawner-master.tar.gz /etc/dockerspawner-master.tar.gz
RUN /opt/conda/bin/pip install /etc/dockerspawner-master.tar.gz

Command to launch jupyterhub as a docker service:

docker service create \   
  --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \    
  --mount type=bind,src=/etc/jupyterhub-swarm,dst=/srv/jupyterhub \    
  --name jupyterhub \    
  --network host \    
  --constraint 'node.role == manager' \    
  --detach=true \    
sangramga/jupyterhub-swarm:latest

I tried over-riding get_ip_and_port() method, but still the jupyterhub service is not able to detect the notebook container. I am unable to pinpoint host_ip since the users Notebook containers will be Spawning in Multiple hosts.

Find my jupyterhub_config.py file.

import os

c.JupyterHub.authenticator_class = 'hashauthenticator.HashAuthenticator'
c.HashAuthenticator.secret_key = 'secret'  # Defaults to ''
c.HashAuthenticator.password_length = 6          # Defaults to 6
c.HashAuthenticator.show_logins = True            # Optional, defaults to False
## The public facing port of the proxy
c.JupyterHub.port = 8000
## The public facing ip of the whole application (the proxy)
c.JupyterHub.ip = '0.0.0.0'
## The ip for this process
c.JupyterHub.hub_ip = '0.0.0.0'
#  Defaults to an empty set, in which case no user has admin access.
c.Authenticator.admin_users = {'tcupadmin'}
c.Authenticator.whitelist = {'notebook2','tcupadmin'}
c.JupyterHub.admin_access = True
network_name = "host"

from jupyterhub.utils import random_port
from tornado import gen
from dockerspawner import SwarmSpawner
from jupyter_client.localinterfaces import public_ips

class custom_spawner(SwarmSpawner):
    @gen.coroutine
    def get_ip_and_port(self):
        # because of the 'network_mode': 'host' settings above, the spawned containers
        # bind to the host, so to access them, we need to use the ip of the host
        return self.host_ip, self.port
@gen.coroutine
    def start(self, *args, **kwargs):
        self.port = random_port()
        # start the container
        ret = yield SwarmSpawner.start(self, *args, **kwargs)
        return ret

c.JupyterHub.spawner_class = custom_spawner
c.SwarmSpawner.network_name = "host"
c.SwarmSpawner.extra_host_config = {'network_mode': network_name}

notebook_dir = os.environ.get('NOTEBOOK_DIR') or '/home/jovyan/work'
c.SwarmSpawner.notebook_dir = notebook_dir
c.SwarmSpawner.image = 'jupyter/base-notebook:latest'
c.SwarmSpawner.debug = True

Following are Error Logs for jupyterhub Docker Service, when user tcupadmin is spawned

jupyterhub.1.lkm1x6ygr640@ip-172-30-0-173    | [E 2018-07-18 12:06:42.216 JupyterHub log:158] 500 GET /hub/user/tcupadmin/ ([email protected]
4.134) 10142.89ms
jupyterhub.1.lkm1x6ygr640@ip-172-30-0-173    | [W 2018-07-18 12:07:01.142 JupyterHub user:504] tcupadmin's server never showed up at http://127
.0.0.1:57446/user/tcupadmin/ after 30 seconds. Giving up
jupyterhub.1.lkm1x6ygr640@ip-172-30-0-173    | [W 2018-07-18 12:07:01.174 JupyterHub swarmspawner:121] Service jupyter-tcupadmin not found

After inspecting the user's notebook Service.
docker service inspect jupyter-tcupadmin shows the following output

"Endpoint": {              
           "Spec": {              
               "Mode": "vip",     
               "Ports": [         
                   {              
                       "Protocol": "tcp",                             
                       "TargetPort": 57446,                           
                       "PublishMode": "ingress"                       
                   }              
               ]                  
           },                     
           "Ports": [             
               {                  
                   "Protocol": "tcp",                                 
                   "TargetPort": 57446,                               
                   "PublishedPort": 30000,                            
                   "PublishMode": "ingress"                           
               }                  
           ],                     
           "VirtualIPs": [        
               {                  
                   "NetworkID": "hlemj2afn0ndkl1jny3lgpa2v",          
                   "Addr": "10.255.2.9/16"                            
               },                 
               {                  
                   "NetworkID": "wplqnqrchc6f49tntpukc7i6b"           
               }                  
           ]                      
       }

[Blizzke]

I'm currently experiencing a similar problem and I was looking into just leaving the notebook on the regular hub network. Like this the proxying works. After the spawn I could use docker network connect to simply provision a second network interface in the container with the public IP. Trying to figure out how to run a command after a notebook is spawned at this point, perhaps I should create an issue here.