Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi! I Deployed Clearml Server Along With Jupyterhub On Azure K8S (Aks). The Way It Works Is That Every User Is Assigned A New Pod That Is Spawned With A Docker Image Of A Choice (One Of Them With Clearml Sdk Installed). I Managed To Configure Most Of The

Hi! I deployed clearml server along with jupyterhub on Azure K8s (AKS). The way it works is that every user is assigned a new pod that is spawned with a docker image of a choice (one of them with clearml SDK installed). I managed to configure most of the things however I get an error that I am not sure how to resolve.
Clearml repository detection throws 404 client error as it unable to access jupyter server and because of that I get wrong path to my notebook "error_notebook_not_found.py". I initialized git repository and it is properly detected however the script is not. I would appreciate any help, excuse me my ignorance if it is something obvious 😉 .

  
  
Posted 3 years ago
Votes Newest

Answers 14


Hi AgitatedDove14 . I'm just writing to explain what was the problem. Basically our setup - jupyterhub on k8s with kubespawner that was spawning a pod for each single user notebook, uses docker images that are based on jupyter/docker-stacks.

The problem was that the token for jupyterhub api was not propagated to the spawned pod so whenever clearml was trying to access jupyter/user/api/sessions endpoint it would be redirected for authorization to jupyterhub api and then fail due to the lack of required token. The other problem was that the runtime directory for jupyter was defaulting to a shared folder so whenever there were more than two single servers pods, the list_running_servers() function was failing as the shared file could not be loaded. The info from list_running_server is used by clearml to know where to send requests to and as it could not read the file - it failed.
Our solution was to basically create a new runtime directory for jupyter but this time not in a shared folder and point jupyter towards it with an env variable. And the token for jupyterhub_api is later added by directly modifying the nbserver-x.json file in runtime directory. (The token is passed from jupyterhub to single-user servers as an env variable "JUPYTERHUB_API_TOKEN").
So as you can see the issue is on the jupyterhub side.

  
  
Posted 3 years ago

Thanks GreasyPenguin66 ! please keep us updated 🙂

  
  
Posted 3 years ago

from your jupyterlab can you do:
!curl

  
  
Posted 3 years ago

It works as well . As for rebuilding the image I was not a root nor a sudoer so I had either to rebuild docker image and set it to root or to install the package while rebuilding 😉

  
  
Posted 3 years ago

That were my thoughts too. But the jupyter/base_notebook from docker stacks that they recommend to use and from which my image inherits did not include the token in the jupyter lab run command. I don't know whether it was a bug or an intentional choice, however I was either going to change the base image, or to add a token in a postStart hook. I decided to go with the second option 😉

  
  
Posted 3 years ago

Sry I had to rebuild my image to install curl. Yes the command works.

  
  
Posted 3 years ago

yes

  
  
Posted 3 years ago

Hi GreasyPenguin66
So the way clearml can store your notebook is by using the jupyter-notebook rest api. It assumes, that it can communicate with it as the kernel is running on the same machine. What exactly is the setup? is the jupyter-lab/notebook running inside the docker? maybe the docker itself is running with some --network argument ?

  
  
Posted 3 years ago

GreasyPenguin66 Nice !!!
Very cool setup, and kudos on making it work with multiple users!
Quick question, shouldn't the JUPYTERHUB_API_TOKEN env variable be enough to gain access to the server? Why did you need to add it to the 'nbserver-x.json' as well?

  
  
Posted 3 years ago

But they are all running inside the same pod, correct ?

  
  
Posted 3 years ago

I'm sorry i was wrong. Niether of the commands give positive response. I actually get 404 page ... Sry I assumed I got a lot of data so it meant it was ok. But now I read into it

  
  
Posted 3 years ago

Thanks GreasyPenguin66
How about:
!curlBTW, no need to rebuild the docker, next time you can always do !apt update && apt install -y <package here> 🙂

  
  
Posted 3 years ago

Anyway thank you so much for the help it seems like not a problem connected with clearml. I ll post my solution once I solve the problem 😉

  
  
Posted 3 years ago

Yes. So jupyter lab is run as a tini command inside a pod. I attach a ps aux command from my pod

  
  
Posted 3 years ago
768 Views
14 Answers
3 years ago
one year ago
Tags
Similar posts