Integrating native-proxy

[jwindgassen]

Hey everyone.

I was recently made aware, of the desire to be able to create standalone proxies, similar to how it is done in jhsingle-native-proxy (see #1).
This would be immensely advantageous for us, so I started porting the code here recently.

There is still a lot to do and I needed to remove/comment out a few of the original features, but it is already fundamentally working as is. I am opening this PR to let you know of this and get an opinion on a few bits here and there. I will continue to improve on it in the next weeks.

In the meantime, any comments and ideas are welcome :)

[ryanlovett]

Hi @jwindgassen , thanks for this contribution!

Is the idea to replace jhsingle-native-proxy (so that code can be shared), or will that project continue on? If the latter then could there still be duplication of code? (will this need to periodically get resynced with jhsingle-native-proxy?) Or should shared code be factored out into something separate for each project to use?

If there is documentation on using this, could that be integrated as well?

[jwindgassen]

That is a good question. I do not know about the current state of the original project. The newest commit was already made a few months ago, but I do not think it is abandoned. I will contact the original developer soon and ask him about the current status and his opinion on this.

But the proxies used by it are almost identical to the proxies here, and (I think) they originally were a copy of an old version of the ones in this package.
IMHO, the best would either be completely merging it into here, or making JSP a dependency of jhsingle-native-proxy and importing the Proxies there. But that is, of course, not my decision to make but one of the original developer.
Furthermore, jhsingle-native-proxy does not support Unix Sockets, which we would love to use.
But since both projects have a lot of similarity, keeping them completely separate from one another is a lot of redundancy.

The Documentation for it is currently in the ReadMe, but could be added to the docs here, given that we decide to add this feature.

[aktech]

This sounds like a great idea. I would suggest to create some kind of checklist of the features that are ported from jhsingle-native-proxy and what's not during the course of this pull request (or a series of pull requests).

We're using jhsingle-native-proxy heavily in jhub-apps would love to move to just using jupyter-server-proxy for everything we can, and having a feature parity checklist will help us understand, what we'll be having/missing with this port.

[jwindgassen]

@ryanlovett I talked to the original developer. He welcomes the merging of his feature into Jupyter Server Proxy. There are currently no further plans for jhsingle-native-proxy, but it will persist (for now), as there are a few custom additions, which I currently do not plan to integrate here.

@aktech Sure. Here is a list with the features I have currently ported or plan to do so:

• Launching standalone proxies from CLI
• Automatically enable JupyterHub Authentication when we are spawned by one
• Send Activity Notifications to JupyterHub
• Ensure usability with Unix-Sockets
• Ensure authenticated Access via JupyterHub
• Allow customization of environment and mappath
• Progressive Proxies (Responsive Proxy for other Stream Types #502)
• Docs & Tests (maybe even with JupyterHub?)
• (If desired) Re-add pulling of Git Repository and venv/conda activation?

That's at least everything I can think of now. If you have other ideas or require something more, let me know and I will add it to the list :)

[ryanlovett]

@jwindgassen Thanks very much for asking! It sounds like this has the potential to consolidate development in the future.

[jwindgassen]

@ryanlovett @aktech I have been working on the feature over the last few weeks, and I am now happy to announce, that the code is working 🎉
I tested it with my own TLJH instance, and it also works with our production JupyterHub Setup. I also verified that authentication with JupyterHub works.
There are a few small things I will probably improve in the future, but I am quite happy with the state of the code as it is now.

I'm welcoming anyone to test the feature on their JupyterHub instance for testing. Please let me know about any problems or errors you encounter when doing so 🙂

How to use

For testing, I like to use voila. The command to execute might look like this:
jupyter standaloneproxy --debug -- voila --no-browser --port={port} /path/to/notebook.ipynb"

What else do you need from my side, besides the code, to be satisfied with this PR? I am currently writing a page for the docs, which I will commit soon.
I was also planning to write a few tests for the feature, but I will have to look into that later.

[ryanlovett]

@jwindgassen Thanks for the update! I'll try to test this week locally.

How might a hub administrator typically configure use of this feature? For example would they set every user to launch voila via standalone proxy from c.Spawner.cmd instead of jupyterhub-singleuser?

Is the intent of standalone to essentially re-use the hub's auth, spawner, user storage, etc. but limit what apps users can invoke because it specifies just the one? (since jupyter server + jupyter-server-proxy enables users to launch an arbitrary number of proxied apps)

[jwindgassen]

In essence, yes.
With the standalone proxy feature, admins/developers of the JupyterHub can give the users of the Hub access to other applications instead of just Notebook/JupyterLab.
While we can achieve the same with the current jupyter-server-proxy via the button in the Launcher, this is often an indirection when you only want to access e.g. RStudio.
By using the SuperviseAndProxyHandler directly, without attaching it to a jupyter-server, we can (re)use the Authentication, User Model, Proxy, etc. JupyterHub provides. Secondly, it is also probably easier to integrate new web apps into an already existing cloud environment this way, than to manually setup everything yourself.

But since you need to overwrite c.Spawner.cmd or modify the behavior of the Spawner in a subclass, new Apps can only be added by the administrators. This gives users access to different apps on the JupyterHub landing page, similar to Open OnDemand. The Spawner then needs to switch between the launch commands accordingly.
In our case, we are planning to provide RStudio, XpraHTML (Remote Desktop), MATLAB, NEST-Desktop, and more to come in the future, using this feature.

[ryanlovett]

Fascinating, thanks!

More of an aside, but how are you customizing the spawner to launch the different applications?

Edit: oh, is it jhub-apps as mentioned earlier?

[jwindgassen]

No. We have created our own custom Spawner. But is similar to this. We have overwritten the start method, which will submit a new Job to our cluster. And inside the start script, we start jupyterhub singleuser at the end.

But now you mention it, jhub-apps might synergyze quite well with the standalone feature.

[manics]

The command line arguments are quite complicated, e.g. having to parse maps.

The alternative is more work, but if we were to refactor ServerProcess to be a Traitlets Configurable

jupyter-server-proxy/jupyter_server_proxy/config.py

Lines 31 to 49 in 41e588a

	ServerProcess = namedtuple(
	"ServerProcess",
	[
	"name",
	"command",
	"environment",
	"timeout",
	"absolute_url",
	"port",
	"unix_socket",
	"mappath",
	"launcher_entry",
	"new_browser_tab",
	"request_headers_override",
	"rewrite_response",
	"update_last_activity",
	"raw_socket_proxy",
	],
	)

I think we could take advantage of Traitlets ability to automatically create all CLI args, and we'd have the benefit of being able to configure jupyter-standaloneproxy using an arbitrarily complex file.

What do you think? I'm happy to investigate converting ServerProxy to Traitlets.

[manics]

There's a mismatch between the options supported by Jupyter Server Proxy
https://github.com/jupyterhub/jupyter-server-proxy/blob/v4.4.0/jupyter_server_proxy/config.py#L34-L47
and those available here. Obviously some options aren't relevant, but for example raw_socket_proxy is.

[jwindgassen]

Thanks for the suggestion, I really like that idea. This should also solve the issue with keeping CLI Arguments up with any new options added to the proxies. I will look into it :)

[manics]

I've made a start in #507

[jwindgassen]

@manics

So I used your branch to create the CLI via traitlets. I'm no expert with traitlets, but I managed to get it working: jwindgassen@2d9eb5b

I had to play around with aliases and the extra_args a bit, but I wanted to keep the CLI reasonably unchanged to before.
Getting the command, which was previously a positional argument, was a bit tricky, since I couldn't find a proper way to add a positional argument to the Parser traitlets create. It was possible by overwriting _create_loader(), but not really pretty. Using extra_args instead works, but we now lack a proper explanation of command when using --help.

If you have a better idea or otherwise comments on my changes over there, let me know! :)

[manics]

Sorry for the delay. Since this is a new addition to Jupyter server proxy I think we should prioritise long term maintenance over just replicating the previous CLI- if there's a better way to do things we can use it as an opportunity to refactor.

We also don't need to do everything in one go, for example it's fine to initially focus on creating a functional standalone proxy along that only supports standard traitlets configuration, and add additional flags in a follow-up PR.

[jwindgassen]

I'm fine with how the CLI looks right now, so I'm happy to switch to this once your PR has been accepted.

I am also almost done with Tests and Docs, they should be ready by the end of the week.

[aktech]

Hey @jwindgassen Thanks a lot for working on this and for the ping. I'll play with it this week to provide some feedback.

[jwindgassen]

Ok, I have now also added Docs and Tests for the new feature.

Writing proper tests for Login and Activity is a bit more difficult, since I would need to spawn a JupyterHub instance to gain full access to its API. For now, I limited it to only testing our code, since the classes I import from JupyterHub are tested over there.

I also added a section to the docs, mostly targeted to developers, where I explain the different sections of the code and what features I needed to implement to make this work smoothly.

If you think we need more tests for specific cases or want something to be explained in the docs in more detail, let me know.