Basically what I want is a
but with a docker container running JupyterHub instead of JupyterLab.
I missed that 🙂
The idea of
clearml-session is to launch a container with jupyterlab (or vscode) on a remote machine, and connect the users machines (i.e. the machine executed the
clearml-session CLI) directly into the container.
Pleacing the jupyterlab with JupyterHub will be meaningless here, becuase the idea it spins an instance (container) per user, no?! What am I missing?
Mostly the configurabilty of
clearml-session and how it was designed. Jupyterhub spawns a process at :8000 which we had to port foreward by hand, but spawning new docker containers using
jupyterhub.Dockerspawner and connecting them to the correct network (the hub should talk to them without
--network host ) seem too difficult or even impossible.
Oh, and there was no JupyterHub stdout in the console output on clearml server, it shows the jupyterlab's output by default
That makes total sense.
So right now you can probably use clearml-session to spin a session in any container, add the jupyterhub to the requirements like so:
clearml-session --packages jupyterhubThen ssh + run jupyerhub + tunnel port?
ssh roo@IP -p 10022 -L 6666:localhost:6666 $ jupyterhubWould that work?
Maybe it is better to add an option to use jupyterhub instead of jupyterlab ?
I don't think I expressed myself well 😅
My problem is I don't know how to run a jupyterhub Task. Basically what I want is a
clearml-session but with a docker container running JupyterHub instead of JupyterLab.
Do I write a Python script? If yes, how can I approach writing it? If not, what are the altenatives?
It's more or less here:
I think that just replacing the package would be enough (I mean you could choose hub/lab, which makes sense to me)
Ohh RotundHedgehog76 this implies a single jupyter hub with multiple uses, is that correct ?
(if this is the case, then yes, clearml-session is definitely not the correct solution, I would look for a helm chart for jupyter hub)
I've never worked with JupyterHub and have little experience with notebooks. What does it do in relation to notebooks?
AgitatedDove14 Well, we have gotten relatively close to the goal, i suppose you wouldn't have to do a lot of work to support it natively
CostlyOstrich36 jupyterhub is a multi-user server, which allows many users to login and spawn their own jupyterlab instances (with custom dependencies, data etc) for runing notebooks
AgitatedDove14 no errors, because I don't know how to start 😅 I am just exploring if anyone did this before I get my hands dirty
I succeeded with your instructions, so thank you!
However, we concluded that we don't want to run it through ClearML after all, so we ran it standalone.
But, I'll update you if we ever run it with ClearML so you could also provide it
I think it should work out of the box, I mean at the end both spin jupyter notebooks, which is what clearml interacts with. Are you getting any errors?
No worries, basically they are independent, spin your JupyerHub , then every user will have to set their own credentials on the JupyterLab instance they use. Maybe there is a way to somehow connect a specific OS environment user->JupyterLab in JupyterHub, that would mean users do not have to worry about credntials. wdyt?
You are not missing nothing, it is what we would like to have, to allow multiple people have their own notebook servers. We have multiple people doing different experiments, and JupyterHub would be their "playground" environment
Thanks, I'll try it out first thing Monday and I'll let you know 😃
That's only a part of a solution.
You'd also have to allow specifying
jupyterhub_config.py , mounting it inside a container at a right place, mounting the docker socket in a secure manner to allow spawning user containers, connecting them to the correct network (
--host won't work), persisting the user database and user data...
Hey AgitatedDove14 ,
This sort of works but not quite. The process runs sucessfully (i can attach the container, ping thr JPH etc) however the portforwarding fails.
When I do the port forward on my own using
ssh -L also seems to fail for jupyterlab and vscode, too, which i find odd
All in all, seems like it will be fairly easy to add JupyterHub to clearml-session, and that would solve your issue, no?
(and it seems from implementation perspective, this will not be a lot of work)
we concluded that we don't want to run it through ClearML after all, so we ran it standalone
out of curiosity, what was the conclusion and why?
We've sucessfully deployed it without helm with custom made docker-compose and makefiles 😄
Yes, that is correct
When I do the port forward on my own using
also seems to fail for jupyterlab and vscode, too, which i find odd
The only external port exposed is the SSH one 10022, then the client forwards it locally (so you, the user, can always have the same connect i.e. "ssh root@localhost -p 8022")
If you need to expose an additional port , when the clearml-session is running, open another terminal and do:
sh root@localhost -p 8022 -L 10123:localhost:6666This should port-forward the container's internal 6666 port to your machine's 10123