WTF?!

Yea, will do so in 30min

Could you send the end file?

Like this?

And then?

Tried to install cudatoolkit==11.1 manually in this environemnt and got:

Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                                         

UnsatisfiableError: The following specifications were found to be incompatible with each other:



Package xz conflicts for:
python=3.8 -> xz[version='>=5.2.4,<5.3.0a0|>=5.2.4,<6.0a0|>=5.2.5,<5.3.0a0|>=5.2.5,<6.0a0']
Package libstdcxx-ng conflicts for:
python=3.8 -> libstdcxx-ng[version='>=7.3.0|>=7.5.0|>=9.3.0']
cudatoolkit=11.1 -> libstdcxx-ng[version='>=9.3.0']
Package libgcc-ng conflicts for:
cudatoolkit=11.1 -> libgcc-ng[version='>=9.3.0']
python=3.8 -> libgcc-ng[version='>=7.3.0|>=7.5.0|>=9.3.0']
Package __glibc conflicts for:
cudatoolkit=11.1 -> __glibc[version='>=2.17,<3.0.a0']
Package libffi conflicts for:
python=3.8 -> libffi[version='>=3.2.1,<3.3.0a0|>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']
Package ncurses conflicts for:
python=3.8 -> ncurses[version='>=6.1,<6.3.0a0|>=6.1,<7.0a0|>=6.2,<6.3.0a0|>=6.2,<7.0a0']
Package zlib conflicts for:
python=3.8 -> zlib[version='>=1.2.11,<1.3.0a0']
Package python_abi conflicts for:
python=3.8 -> python_abi[version='*|3.8.*',build=*_cp38]
Package sqlite conflicts for:
python=3.8 -> sqlite[version='>=3.30.0,<4.0a0|>=3.30.1,<4.0a0|>=3.31.1,<4.0a0|>=3.32.3,<4.0a0|>=3.33.0,<4.0a0|>=3.34.0,<4.0a0']
Package bzip2 conflicts for:
python=3.8 -> bzip2[version='>=1.0.8,<2.0a0']
Package readline conflicts for:
python=3.8 -> readline[version='>=7.0,<8.0a0|>=8.0,<9.0a0']
Package openssl conflicts for:
python=3.8 -> openssl[version='>=1.1.1a,<1.1.2a|>=1.1.1d,<1.1.2a|>=1.1.1e,<1.1.2a|>=1.1.1f,<1.1.2a|>=1.1.1g,<1.1.2a|>=1.1.1h,<1.1.2a|>=1.1.1i,<1.1.2a|>=1.1.1j,<1.1.2a']
Package tk conflicts for:
python=3.8 -> tk[version='>=8.6.10,<8.7.0a0|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0']
Package pip conflicts for:
python=3.8 -> pip
Package ld_impl_linux-64 conflicts for:
python=3.8 -> ld_impl_linux-64[version='>=2.34']The following specifications were found to be incompatible with your CUDA driver:

  - cudatoolkit=11.1 -> __cuda[version='>=11.1']

Your installed CUDA driver is: 11.2

Let me check

# Python 3.7.10 (default, Feb 26 2021, 18:47:35)  [GCC 7.3.0]

aiostream==0.4.2
attrs==20.3.0
clearml==0.17.4
dm-control==0.0.355168290
dm-env==1.4
furl==2.1.0
future==0.18.2
glfw==2.1.0
gym==0.18.0
humanfriendly==9.1
imageio-ffmpeg==0.4.3
jsonschema==3.2.0
labmaze==1.0.3
lxml==4.6.2
moviepy==1.0.3
orderedmultidict==1.0.1
pathlib2==2.3.5
pillow==7.2.0
proglog==0.1.9
psutil==5.8.0
pybullet==3.0.9
pygame==2.0.1
pyglet==1.5.0
pyjwt==2.0.1
pyrsistent==0.17.3
requests-file==1.5.1
tensorboard==2.4.1
tensorboardx==2.1

# Conda Packages

blas==1.0
bzip2==1.0.8
ca-certificates==2020.10.14
certifi==2020.6.20
cloudpickle==1.6.0
cudatoolkit==11.1.1
cycler==0.10.0
cytoolz==0.11.0
dask-core==2021.2.0
decorator==4.4.2
ffmpeg==4.3
freetype==2.10.4
gmp==6.2.1
gnutls==3.6.13
imageio==2.9.0
jpeg==9b
kiwisolver==1.3.1
lame==3.100
lcms2==2.11
ld_impl_linux-64==2.33.1
libedit==3.1.20191231
libffi==3.3
libgcc-ng==9.3.0
libgfortran-ng==7.3.0
libiconv==1.16
libpng==1.6.37
libstdcxx-ng==9.3.0
libtiff==4.1.0
libuv==1.41.0
llvm-openmp==11.0.1
lz4-c==1.9.3
matplotlib-base==3.3.4
mkl==2020.4
mkl-service==2.3.0
mkl_fft==1.3.0
mkl_random==1.2.0
ncurses==6.2
nettle==3.6
networkx==2.5
ninja==1.10.2
numpy==1.19.2
numpy-base==1.19.2
olefile==0.46
openh264==2.1.1
openssl==1.1.1j
pip==21.0.1
pyparsing==2.4.7
python==3.7.10
python-dateutil==2.8.1
python_abi==3.7
torch==1.8.0
pywavelets==1.1.1
pyyaml==5.3.1
readline==8.1
scikit-image==0.17.2
scipy==1.6.1
setuptools==52.0.0
six==1.15.0
sqlite==3.33.0
tifffile==2020.10.1
tk==8.6.10
toolz==0.11.1
torchaudio==0.8.0
torchvision==0.9.0
tornado==6.1
typing_extensions==3.7.4.3
wheel==0.36.2
xz==5.2.5
yaml==0.2.5
zlib==1.2.11
zstd==1.4.9

Hmm, you are correct
Which means this is some conda issue, basically when installing from env file, conda is not resolving the correct pytorch version 😞
Not sure why... Could you try to upgrade conda ?

The problem is that clearml installs 

cudatoolkit=11.0

 but 

cudatoolkit=11.1

 is needed.
You suggested this fix earlier, but I am not sure why it didnt work then.

Hmm , could you test with the clearml-agent 0.17.2 ? making surethis actually solves the problem

And this works fine.

Okay found it 🙂 it returns 11020 instead of 112

Yep, this install PyTorch CPU

Installs CPU

I will try a minimal version now

I just started a task from this environment and it fails on the agent.

Thanks! Tomorrow is great, I'll put the wheel here 🙂

Did not happen with conda 4.9.2

You suggested this fix earlier, but I am not sure why it didnt work then.

This is the venc

My driver says "CUDA Version: 11.2" (I am not even sure this is correct, since I do not remember installing code in this machine, but idk) and there is no pytorch for 11.2, so maybe it fallbacks to cpu?

@<1523701868901961728:profile|ReassuredTiger98> what are you getting with:

nvidia-smi

And here:

ls -la /usr/local/

channels:
- pytorch
- conda-forge
- defaults
dependencies:
- cudatoolkit~=11.1.1
- pytorch~=1.8.0

Works fine

But I do not have anything linked correctly since I rely in conda installing cuda/cudnn for me

I mean the version which it bases the PyTorch installation on.

Okay.

Hi @<1523701868901961728:profile|ReassuredTiger98> when you get to it...
please download the wheel, then install it with

pip3 install -U clearml_agent-0.17.3rc0-py3-none-any.whl

Then run the daemon with the additional --debug argument, basically:

clearml-agent --debug daemon --foreground ...

Once the agent is running please send the Task's log from your console 🙂

ok, thanks!

Do you know how I can get this version?

I just tried to envrionment setup steps that clearml-agent is doing locally, but with my environment.yml instead of the one that clearml generates.