btw: why is agent.package_manager and agent attribute. Imo it does not make sense because conda can install pip packages, but pip cannot install conda packages which can lead to install failures, right?
Sorry, env file for conda, the one you are using to install
The problem is that clearml installs
cudatoolkit=11.0
but
cudatoolkit=11.1
is needed.
You suggested this fix earlier, but I am not sure why it didnt work then.
Hmm , could you test with the clearml-agent 0.17.2 ? making surethis actually solves the problem
Okay this is very close to what the agent is building:
Could you start a new conda env,
then install cudatoolkit=11.1
then run:
conda env update -p <conda_env_path_here> --file the_env_yaml.yml
Just tested again. The ordering definitly matters.
Hmm, you are correct
Which means this is some conda issue, basically when installing from env file, conda is not resolving the correct pytorch version 😞
Not sure why... Could you try to upgrade conda ?
Okay this seems correct:
pytorch=1.8.0=py3.7_cuda11.1_cudnn8.0.5_0
I can't seem to find what's the diff between the two.
Give me a second let me check if I can reproduce it somehow.
I do not have a global cuda install on this machine. Everything except for the driver is installed via conda.
fyi: NVIDIA-SMI 460.56 Driver Version: 460.56 CUDA Version: 11.2
I guess that has nothing to do with the diff version, right ?
Interesting: This command failes (with an error similar to the one I posted above) in conda version 4.7.12 but runs just fine in version 4.9.2: conda create --name test-pytorch python=3.8 cudatoolkit=11.1 -c conda-forge
For now I can tell you that with conda_freeze: true it fails, but with conda_freeze: false it works!
But I do not have anything linked correctly since I rely in conda installing cuda/cudnn for me
Yes I think the difference is running conda install with arguments vs conda install with env file...
conda_channels: ["defaults", "conda-forge", "pytorch", ]
Damn, okay I'll make sure we fix the order.
Could you verify the ~= works as intended (if the order id correct)
# Python 3.7.10 (default, Feb 26 2021, 18:47:35) [GCC 7.3.0]
aiostream==0.4.2
attrs==20.3.0
clearml==0.17.4
dm-control==0.0.355168290
dm-env==1.4
furl==2.1.0
future==0.18.2
glfw==2.1.0
gym==0.18.0
humanfriendly==9.1
imageio-ffmpeg==0.4.3
jsonschema==3.2.0
labmaze==1.0.3
lxml==4.6.2
moviepy==1.0.3
orderedmultidict==1.0.1
pathlib2==2.3.5
pillow==7.2.0
proglog==0.1.9
psutil==5.8.0
pybullet==3.0.9
pygame==2.0.1
pyglet==1.5.0
pyjwt==2.0.1
pyrsistent==0.17.3
requests-file==1.5.1
tensorboard==2.4.1
tensorboardx==2.1
# Conda Packages
blas==1.0
bzip2==1.0.8
ca-certificates==2020.10.14
certifi==2020.6.20
cloudpickle==1.6.0
cudatoolkit==11.1.1
cycler==0.10.0
cytoolz==0.11.0
dask-core==2021.2.0
decorator==4.4.2
ffmpeg==4.3
freetype==2.10.4
gmp==6.2.1
gnutls==3.6.13
imageio==2.9.0
jpeg==9b
kiwisolver==1.3.1
lame==3.100
lcms2==2.11
ld_impl_linux-64==2.33.1
libedit==3.1.20191231
libffi==3.3
libgcc-ng==9.3.0
libgfortran-ng==7.3.0
libiconv==1.16
libpng==1.6.37
libstdcxx-ng==9.3.0
libtiff==4.1.0
libuv==1.41.0
llvm-openmp==11.0.1
lz4-c==1.9.3
matplotlib-base==3.3.4
mkl==2020.4
mkl-service==2.3.0
mkl_fft==1.3.0
mkl_random==1.2.0
ncurses==6.2
nettle==3.6
networkx==2.5
ninja==1.10.2
numpy==1.19.2
numpy-base==1.19.2
olefile==0.46
openh264==2.1.1
openssl==1.1.1j
pip==21.0.1
pyparsing==2.4.7
python==3.7.10
python-dateutil==2.8.1
python_abi==3.7
torch==1.8.0
pywavelets==1.1.1
pyyaml==5.3.1
readline==8.1
scikit-image==0.17.2
scipy==1.6.1
setuptools==52.0.0
six==1.15.0
sqlite==3.33.0
tifffile==2020.10.1
tk==8.6.10
toolz==0.11.1
torchaudio==0.8.0
torchvision==0.9.0
tornado==6.1
typing_extensions==3.7.4.3
wheel==0.36.2
xz==5.2.5
yaml==0.2.5
zlib==1.2.11
zstd==1.4.9
Is ther a way to see the contents of /tmp/conda_envaz1ne897.yml ? Seems to be deleted after the task is finihsed
Do you know how I can get this version?
ReassuredTiger98 in the UI can you see it in the "installed packages" section under the Execution Tab ?
I installed my local conda environment from an environment.yml without issues, so maybe clearml makes some changes that leads to conflicts which finally leads to the cpu-version install.