within the docker image and the conda environment

SuccessfulKoala55 SweetBadger76 hey guys i tried to run this line task.set_base_docker("<image> -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/conda/envs/rapids/bin/python -e CLEARML_AGENT__AGENT__PACKAGE_MANAGER_ TYPE=conda -e C LEARML_AGENT__VENV_DIR=/opt/conda/envs") but it is throwing a conda DirectoryNotACondaEnvironmentError, expecting a python 3.8 environment. Am i missing something out here?

hmmm unfortunately it isn't as straightforward...installing it on python-pip throws this exception - Exception: Please install cuml via the rapidsai conda channel. See https://rapids.ai/start.html for instructions.

can you share the entire log with the error from the agent?

ahhh okay the logs are in a closed environment but i will try to extract what i can 🙏

Hey SuccessfulKoala55 , i figured a workaround to the problem and just wanted to close the loop. Rapids requires c++ code to be integrated into their package and also auxiliary packages inside their prebuilt image and the pip ecosystem currently doesn't support their requirements https://medium.com/rapids-ai/rapids-0-7-release-drops-pip-packages-47fc966e9472 (hence the need to use conda). Instead of trying to run conda with clearml-agent i figured it might be possible to pass the PYTHONPATH=<conda path> env variable to direct python to look for packages in the conda environment instead. It works as intended and i'm satisfied with the workaround for now.

And how did you create the conda venv in the docker?

Well, I'm actually not sure if the conda code handles this venv correctly, let me check

it was pre-built by rapidsai themselves

the prefix for the agent is CLEARML_AGENT__

It seems this flag is only meant to handle pip, not conda 😞

for instance
export CLEARML_AGENT__AGENT__PACKAGE_MANAGER_ TYPE=conda && clearml-agent daemon --queue my queue

You can set any agent setting from either the clearml.conf file or environment variables

Also, you have a typo:
task.set_base_docker("<image> -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/conda/envs/rapids/bin/python -e CLEARML_AGENT__AGENT__PACKAGE_MANAGER_ TYPE=conda -e C LEARML_AGENT__VENV_DIR=/opt/conda/envs")
Should be
task.set_base_docker("<image> -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/conda/envs/rapids/bin/python -e CLEARML_AGENT__AGENT__PACKAGE_MANAGER__TYPE=conda -e CLEARML_AGENT__VENV_DIR=/opt/conda/envs")

yup yup the code runs good locally

SuccessfulKoala55 Yes, i believe if it's within the .set_base_docker(...) method i should be able to? Is there a specific env variable i can set?

This is an example of how to do it in general: https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_config#dynamic-environment-variables

SweetBadger76 Slack got you... should be CLEARML_AGENT__AGENT__PACKAGE_MANAGER__TYPE=conda

Same error without the VENV_DIR variable. oops sorry typo it was already double spaced

task.set_base_docker("<image> -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/conda/envs/rapids/bin/python -e CLEARML_AGENT__AGENT__PACKAGE_MANAGER__TYPE=conda")

oops sorry i found the repo in the .clearml/venv-builds/ folder but but im not sure why the remainder of the code isn't executed still

What if you do not provide CLEARML_AGENT__VENV_DIR=/opt/conda/envs ?

Yes actually, i'm trying to access cudf/cuml libraries on rapids and the official guide insist that these libraries within the image has to be used with conda

Ahh okay this was the specific replication of the environment
task.set_base_docker("rapidsai/rapidsai-dev:21.10-cuda11.0-devel-ubuntu18.04-py3.8 -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/conda/envs/rapids/bin/python -e CLEARML_AGENT__AGENT__PACKAGE_MANAGER__TYPE=conda")

ideally the code should be able to import cuml

and can't you let the agent install it itself?

mmm are there any methods to approach this (toggling between pip and conda mode) at the code level? i'm actually not allowed to reconfigure the agents as a developer-user.

Is there any specific reason you must use conda?

and are you sure the venv inside is good?

Can you use environment variables like you've set CLEARML_AGENT_PIP_VENV_INSTALL?

locally meaning from within the docker container?

Have you tried try to set your agent in conda mode ( https://clear.ml/docs/latest/docs/clearml_agent#conda-mode ) ?