Hi CostlyOstrich36 , I'm using clearml-agent==1.3.0.
I ran the training script on my local machine without using an agent (just logging with clearml) and everything worked fine.
I think it's trying to install the packages in /tmp/user/ because I'm inside tmux. Any way I can manually install the environment and then point the agent to it?
Hard to tell 🤔 , I'm using the version of conda that comes with mamba but I don't think it changes anything. It never gave me any problem and it seems the clearml-agent is able to normally use it to install python and torch before failing on open3d. Here's the full output:
All right that made sense so I checked but I still think it's using the right version.
This is the Installed Package section of the experiment I'm trying to enqueue as reported on the ClearML dashboard:# Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:18) [GCC 10.3.0] PyYAML == 5.4.1 clearml == 1.6.2 matplotlib == 3.5.2 numpy == 1.23.1 open3d == 0.15.2 plotly == 5.9.0 scipy == 1.9.0 torch == 1.10.2 tqdm == 4.64.0 .So it knows it has to use python 3.8 .
Also th...
Sorry to further bother you, but do you happen to have any idea on how to solve this? https://clearml.slack.com/archives/CTK20V944/p1659602063081199
Nope it's a linux machine with a xeon gold 5218. Anyway I installed the rc and now it seems to work a bit better: it gets past the torch installation but fails installing open3d.
ERROR: Could not find a version that satisfies the requirement open3d==0.15.2 (from -r /tmp/user/1021449697/cached-reqse9v2emqx.txt (line 4)) (from versions: 0.10.0.0, 0.11.0, 0.11.1, 0.11.2, 0.12.0, 0.13.0)
Technically 0.15.2 shouldn't give any problem. I have a conda environment on the same machine with all...
SuccessfulKoala55 CostlyOstrich36 How can I solve it? If I run conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch in an environment, it installs it without any problem. I guess it's trying to install it with pip instead than conda even though I had set conda in the clearml.config file.
Since automatically building an environment is often problematic it would be a great feature to point the agent to a pre-built environment with all the packages it will need.
All right that sounds useful thanks! I'm not sure that Task.init() should handle reproducibility by default but maybe it may just be personal taste