Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello, I'M Using A Virtual Environment Inside My Jupyterhub Server Along With Clearml. Whenever I Create Any Task The "Uncommitted Changes" Are The Contents Of

Hello, I'm using a virtual environment inside my Jupyterhub server along with ClearML.
Whenever I create any task the "uncommitted changes" are the contents of ipykernel_launcher.py , is there a way to make ClearML recognize that I'm running inside a venv?

  
  
Posted one year ago
Votes Newest

Answers 17


When running inside jupyter, ClearML basically stored the notebook itself inside the uncommitted changes, so that you can run this again yourself

  
  
Posted one year ago

When working outside of my venv I can see my jupyter notebook inside "uncommitted changes" and also as an artifact. When inside a venv I dont see it as an artifact and all I see under "uncommited changes" is the contents of ipykernel_launcher.py

  
  
Posted one year ago

seems like an issue when using ipykernel...
steps to reproduce:|
virtualenv my_env
source my_env/bin/activate
pip install ipykernl
python -m ipykernel install --user --name=my_env

then switch the kernel to the new ipykernel to reproduce

  
  
Posted one year ago

When I'm using the default python kernel for notebooks it does store it successfully, however when switching to a venv it doesn't.

  
  
Posted one year ago

Or enqueue a clone of the task for remote execution

  
  
Posted one year ago

I should mention that reporting artifacts and everything else works. I'm just not seeing the changes inside the web ui.

  
  
Posted one year ago

let me correct myself, when switching to a different python kernel it breaks.

  
  
Posted one year ago

@<1535793988726951936:profile|YummyElephant76>

Whenever I create any task the "uncommitted changes" are the contents of

ipykernel_launcher.py

, is there a way to make ClearML recognize that I'm running inside a venv?

This sounds like a bug, it should have the entire notebook there, no?

  
  
Posted one year ago

Hi @<1535793988726951936:profile|YummyElephant76> , what changes are you not seeing?

  
  
Posted one year ago

yeah

  
  
Posted one year ago

So it sounds as if for some reason calling Task.init inide a notebook on your jupyterhub is not detecting the notebook.
Is there anything special about the jupyterhub deployment ? how is it deployed ? is it password protected ? is this reproducible ?

  
  
Posted one year ago

It is deployed on an on premise, secured network that has no access to the outside world.

Is it password protected or something of that nature?

Perhaps we could find a different solution or work around, rather than solving a technical issue.

Solving it means allowing the python code to ask the JupyterLab server for the notebook file

However, once working with ClearML and using a venv (and not the default python kernel),

Are you saying on your specific setup (i.e. OpenShift+helm etc) in some occurrences everything work and on some it does not?

  
  
Posted one year ago

@<1523701205467926528:profile|AgitatedDove14> hey, we found out what was causing that issue
when a new venv is created it does not contain any python libraries, so when ClearML was trying to list the current running jupyter servers (using the Jupyter Notebook python library) it was failing since that library does not exist.
Not sure why there were no warnings or errors regarding it...
We fixed it by running pip install notebook inside the venv, and it worked!
CC: @<1564422819012415488:profile|DisturbedCormorant14>

  
  
Posted one year ago

@<1535793988726951936:profile|YummyElephant76> oh you mean like jupyter server was running, then inside the notebook you would start a new venv, in that venv "notebook" package was missing, hence it failed detecting the notebook ?

  
  
Posted one year ago

Hmm Okay, I think the takeaway is that we should print "missing notebook package" 🙂

  
  
Posted one year ago

Hi @<1523701205467926528:profile|AgitatedDove14> , I'm RoseGold's coworker.
The issue still occurs. I wil try to provide you with the necessary information.
We are using a bitnami helm release of jupyterhub for deployement on Openshift.
the version is 3.1.0.
It is deployed on an on premise, secured network that has no access to the outside world.

Perhaps we could find a different solution or work around, rather than solving a technical issue.
The thing is that researcher in our organization sometimes need to use specific versions of specific libraries for their model to run successfully. In order to do that the use venv's with their requirements.
However, once working with ClearML and using a venv (and not the default python kernel), then the notebook will not upload to the ClearML server as an artifact and is not found in storage.

Would really appreciate it if you could help us out

  
  
Posted one year ago

yes, exactly

  
  
Posted one year ago
1K Views
17 Answers
one year ago
one year ago
Tags