Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, I'M Trying To Use Cuml With Clearml And Am Curious If Anyone Has Encountered Running The Rapids Image On A Privately Hosted Clearml Agent Before? I Set Clearml_Agent_Pip_Venv_Install=/Opt/Conda/Envs/Rapids/Bin/Python As A Flag Inside Task.Set_Base

Hi all, I'm trying to use cuml with clearml and am curious if anyone has encountered running the rapids image on a privately hosted clearml agent before? I set CLEARML_AGENT_PIP_VENV_INSTALL=/opt/conda/envs/rapids/bin/python as a flag inside task.set_base_docker("..."). The environment setups successfully but it gets stuck right before the task execution

e.g.
Environment setup completed successfully
Starting Task Execution:

  
  
Posted 2 years ago
Votes Newest

Answers 30


It seems this flag is only meant to handle pip, not conda 😞

  
  
Posted 2 years ago

hmmm unfortunately it isn't as straightforward...installing it on python-pip throws this exception - Exception: Please install cuml via the rapidsai conda channel. See https://rapids.ai/start.html for instructions.

  
  
Posted 2 years ago

oops sorry i found the repo in the .clearml/venv-builds/ folder but but im not sure why the remainder of the code isn't executed still

  
  
Posted 2 years ago

And how did you create the conda venv in the docker?

  
  
Posted 2 years ago

it was pre-built by rapidsai themselves

  
  
Posted 2 years ago

ahhh okay the logs are in a closed environment but i will try to extract what i can 🙏

  
  
Posted 2 years ago

SuccessfulKoala55 SweetBadger76 hey guys i tried to run this line task.set_base_docker("<image> -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/conda/envs/rapids/bin/python -e CLEARML_AGENT__AGENT__PACKAGE_MANAGER_ TYPE=conda -e C LEARML_AGENT__VENV_DIR=/opt/conda/envs") but it is throwing a conda DirectoryNotACondaEnvironmentError, expecting a python 3.8 environment. Am i missing something out here?

  
  
Posted 2 years ago

Same error without the VENV_DIR variable. oops sorry typo it was already double spaced

task.set_base_docker("<image> -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/conda/envs/rapids/bin/python -e CLEARML_AGENT__AGENT__PACKAGE_MANAGER__TYPE=conda")

  
  
Posted 2 years ago

the prefix for the agent is CLEARML_AGENT__

  
  
Posted 2 years ago

and can't you let the agent install it itself?

  
  
Posted 2 years ago

Can you use environment variables like you've set CLEARML_AGENT_PIP_VENV_INSTALL?

  
  
Posted 2 years ago

and are you sure the venv inside is good?

  
  
Posted 2 years ago

Hey SuccessfulKoala55 , i figured a workaround to the problem and just wanted to close the loop. Rapids requires c++ code to be integrated into their package and also auxiliary packages inside their prebuilt image and the pip ecosystem currently doesn't support their requirements https://medium.com/rapids-ai/rapids-0-7-release-drops-pip-packages-47fc966e9472 (hence the need to use conda). Instead of trying to run conda with clearml-agent i figured it might be possible to pass the PYTHONPATH=<conda path> env variable to direct python to look for packages in the conda environment instead. It works as intended and i'm satisfied with the workaround for now.

  
  
Posted 2 years ago

Have you tried try to set your agent in conda mode ( https://clear.ml/docs/latest/docs/clearml_agent#conda-mode ) ?

  
  
Posted 2 years ago

What if you do not provide CLEARML_AGENT__VENV_DIR=/opt/conda/envs ?

  
  
Posted 2 years ago

yup yup the code runs good locally

  
  
Posted 2 years ago

Is there any specific reason you must use conda?

  
  
Posted 2 years ago

mmm are there any methods to approach this (toggling between pip and conda mode) at the code level? i'm actually not allowed to reconfigure the agents as a developer-user.

  
  
Posted 2 years ago

Well, I'm actually not sure if the conda code handles this venv correctly, let me check

  
  
Posted 2 years ago

SweetBadger76 Slack got you... should be CLEARML_AGENT__AGENT__PACKAGE_MANAGER__TYPE=conda

  
  
Posted 2 years ago

locally meaning from within the docker container?

  
  
Posted 2 years ago

can you share the entire log with the error from the agent?

  
  
Posted 2 years ago

Also, you have a typo:
task.set_base_docker("<image> -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/conda/envs/rapids/bin/python -e CLEARML_AGENT__AGENT__PACKAGE_MANAGER_ TYPE=conda -e C LEARML_AGENT__VENV_DIR=/opt/conda/envs")
Should be
task.set_base_docker("<image> -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/conda/envs/rapids/bin/python -e CLEARML_AGENT__AGENT__PACKAGE_MANAGER__TYPE=conda -e CLEARML_AGENT__VENV_DIR=/opt/conda/envs")

  
  
Posted 2 years ago

Yes actually, i'm trying to access cudf/cuml libraries on rapids and the official guide insist that these libraries within the image has to be used with conda

  
  
Posted 2 years ago

Ahh okay this was the specific replication of the environment
task.set_base_docker("rapidsai/rapidsai-dev:21.10-cuda11.0-devel-ubuntu18.04-py3.8 -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/conda/envs/rapids/bin/python -e CLEARML_AGENT__AGENT__PACKAGE_MANAGER__TYPE=conda")

ideally the code should be able to import cuml

  
  
Posted 2 years ago

You can set any agent setting from either the clearml.conf file or environment variables

  
  
Posted 2 years ago

SuccessfulKoala55 Yes, i believe if it's within the .set_base_docker(...) method i should be able to? Is there a specific env variable i can set?

  
  
Posted 2 years ago

within the docker image and the conda environment

  
  
Posted 2 years ago

for instance
export CLEARML_AGENT__AGENT__PACKAGE_MANAGER_ TYPE=conda && clearml-agent daemon --queue my queue

  
  
Posted 2 years ago
672 Views
30 Answers
2 years ago
one year ago
Tags