Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello All

Hello all ๐Ÿ˜Š

Has anyone attempted to use JupyterHub (not JupyterLab!) server with ClearML?

We would like to run jupyterhub and spawn small jupyterlab instances for every user. I have a standalone deployment of that with docker-compose , but I'm kinda stuck with integrating it to ClearML

  
  
Posted 2 years ago
Votes Newest

Answers 22


Basically what I want is aย 

clearml-session

ย but with a docker container running JupyterHub instead of JupyterLab.

I missed that ๐Ÿ™‚

The idea of clearml-session is to launch a container with jupyterlab (or vscode) on a remote machine, and connect the users machines (i.e. the machine executed the clearml-session CLI) directly into the container.

Pleacing the jupyterlab with JupyterHub will be meaningless here, becuase the idea it spins an instance (container) per user, no?! What am I missing?

  
  
Posted 2 years ago

I succeeded with your instructions, so thank you!

However, we concluded that we don't want to run it through ClearML after all, so we ran it standalone.

But, I'll update you if we ever run it with ClearML so you could also provide it

  
  
Posted 2 years ago

Awesome !

  
  
Posted 2 years ago

Hey AgitatedDove14 ,

This sort of works but not quite. The process runs sucessfully (i can attach the container, ping thr JPH etc) however the portforwarding fails.

When I do the port forward on my own using ssh -L also seems to fail for jupyterlab and vscode, too, which i find odd

  
  
Posted 2 years ago

That's only a part of a solution.
You'd also have to allow specifying jupyterhub_config.py , mounting it inside a container at a right place, mounting the docker socket in a secure manner to allow spawning user containers, connecting them to the correct network ( --host won't work), persisting the user database and user data...

  
  
Posted 2 years ago

We've sucessfully deployed it without helm with custom made docker-compose and makefiles ๐Ÿ˜„

  
  
Posted 2 years ago

Yes, that is correct

  
  
Posted 2 years ago

It's more or less here:
https://github.com/allegroai/clearml-session/blob/0dc094c03dabc64b28dcc672b24644ec4151b64b/clearml_session/interactive_session_task.py#L431
I think that just replacing the package would be enough (I mean you could choose hub/lab, which makes sense to me)

  
  
Posted 2 years ago

Ohh RotundHedgehog76 this implies a single jupyter hub with multiple uses, is that correct ?
(if this is the case, then yes, clearml-session is definitely not the correct solution, I would look for a helm chart for jupyter hub)

  
  
Posted 2 years ago

CostlyOstrich36 jupyterhub is a multi-user server, which allows many users to login and spawn their own jupyterlab instances (with custom dependencies, data etc) for runing notebooks

AgitatedDove14 no errors, because I don't know how to start ๐Ÿ˜… I am just exploring if anyone did this before I get my hands dirty

  
  
Posted 2 years ago

Hi RotundHedgehog76
I think it should work out of the box, I mean at the end both spin jupyter notebooks, which is what clearml interacts with. Are you getting any errors?

  
  
Posted 2 years ago

When I do the port forward on my own usingย 

ssh -L

ย also seems to fail for jupyterlab and vscode, too, which i find odd

The only external port exposed is the SSH one 10022, then the client forwards it locally (so you, the user, can always have the same connect i.e. "ssh root@localhost -p 8022")
If you need to expose an additional port , when the clearml-session is running, open another terminal and do:
sh root@localhost -p 8022 -L 10123:localhost:6666This should port-forward the container's internal 6666 port to your machine's 10123

  
  
Posted 2 years ago

Mostly the configurabilty of clearml-session and how it was designed. Jupyterhub spawns a process at :8000 which we had to port foreward by hand, but spawning new docker containers using jupyterhub.Dockerspawner and connecting them to the correct network (the hub should talk to them without --network host ) seem too difficult or even impossible.

Oh, and there was no JupyterHub stdout in the console output on clearml server, it shows the jupyterlab's output by default

  
  
Posted 2 years ago

I've never worked with JupyterHub and have little experience with notebooks. What does it do in relation to notebooks?

  
  
Posted 2 years ago

No worries, basically they are independent, spin your JupyerHub , then every user will have to set their own credentials on the JupyterLab instance they use. Maybe there is a way to somehow connect a specific OS environment user->JupyterLab in JupyterHub, that would mean users do not have to worry about credntials. wdyt?

  
  
Posted 2 years ago

Thanks, I'll try it out first thing Monday and I'll let you know ๐Ÿ˜ƒ

  
  
Posted 2 years ago

we concluded that we don't want to run it through ClearML after all, so we ran it standalone

out of curiosity, what was the conclusion and why?

  
  
Posted 2 years ago

AgitatedDove14 Well, we have gotten relatively close to the goal, i suppose you wouldn't have to do a lot of work to support it natively

  
  
Posted 2 years ago

All in all, seems like it will be fairly easy to add JupyterHub to clearml-session, and that would solve your issue, no?
(and it seems from implementation perspective, this will not be a lot of work)
wdyt?

  
  
Posted 2 years ago

You are not missing nothing, it is what we would like to have, to allow multiple people have their own notebook servers. We have multiple people doing different experiments, and JupyterHub would be their "playground" environment

  
  
Posted 2 years ago

That makes total sense.
So right now you can probably use clearml-session to spin a session in any container, add the jupyterhub to the requirements like so:
clearml-session --packages jupyterhubThen ssh + run jupyerhub + tunnel port?
ssh roo@IP -p 10022 -L 6666:localhost:6666 $ jupyterhubWould that work?
Maybe it is better to add an option to use jupyterhub instead of jupyterlab ?
wdyt?

  
  
Posted 2 years ago

I don't think I expressed myself well ๐Ÿ˜…

My problem is I don't know how to run a jupyterhub Task. Basically what I want is a clearml-session but with a docker container running JupyterHub instead of JupyterLab.

Do I write a Python script? If yes, how can I approach writing it? If not, what are the altenatives?

  
  
Posted 2 years ago
554 Views
22 Answers
2 years ago
one year ago
Tags