okay, I'll make sure we order it correctly

Yes that is exactly what I will make sure we change :)

I try it one more time just to make sure

Quick question: Where again does clearml place the venv? I wanna take a look into it after the task has failed

sure

Okay.

@<1523701868901961728:profile|ReassuredTiger98> what do you have in the clearml.conf under "conda_channels" ?
Is this it ?
None

Or there should be an early error for trying to run conda based tasks on pip agents

Same error.

drwxr-xr-x 10 root root 4096 Jul 31  2020 .
drwxr-xr-x 14 root root 4096 Jul 31  2020 ..
drwxr-xr-x  2 root root 4096 Feb  4 13:52 bin
drwxr-xr-x  2 root root 4096 Jul 31  2020 etc
drwxr-xr-x  2 root root 4096 Jul 31  2020 games
drwxr-xr-x  2 root root 4096 Jul 31  2020 include
drwxr-xr-x  4 root root 4096 Feb  3 13:40 lib
lrwxrwxrwx  1 root root    9 Dez 10 14:29 man -> share/man
drwxr-xr-x  2 root root 4096 Jul 31  2020 sbin
drwxr-xr-x  7 root root 4096 Jul 31  2020 share
drwxr-xr-x  2 root root 4096 Jul 31  2020 src

Still shows CPU version when I run conda list

Thu Mar 11 17:52:45 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.56       Driver Version: 460.56       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    Off  | 00000000:01:00.0 Off |                  N/A |
| 61%   63C    P2   296W / 350W |   8318MiB / 24268MiB |     74%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 3090    Off  | 00000000:21:00.0 Off |                  N/A |
| 30%   29C    P8    20W / 350W |      1MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    133165    C+G   ...s-builds.1/3.7/bin/python     8314MiB |
+-----------------------------------------------------------------------------+

Complete conda log

btw: I also tested the clearml-agent running on a different machine and with python 3.8 and I get the same problems.

Will do!

Ha?!

How does clearml-agent create the conda environment?

@<1523701868901961728:profile|ReassuredTiger98> thank you so much for testing it!

Yep, this install PyTorch CPU

Hi @<1523701868901961728:profile|ReassuredTiger98> when you get to it...
please download the wheel, then install it with

pip3 install -U clearml_agent-0.17.3rc0-py3-none-any.whl

Then run the daemon with the additional --debug argument, basically:

clearml-agent --debug daemon --foreground ...

Once the agent is running please send the Task's log from your console 🙂

Thank you! 🙂

I do not have a global cuda install on this machine. Everything except for the driver is installed via conda.

You suggested this fix earlier, but I am not sure why it didnt work then.

Can you ping me when it is updated in None so I can update my installation?

And the one with the CPU version? is it with "~=" or "="?

Thanks! Tomorrow is great, I'll put the wheel here 🙂

Give me a minute

This is the venc

Sorry, env file for conda, the one you are using to install

The ordering of the channels seems to matter!