The package is just subdir by the way. So it should not be in installed packages anyways, right?

ok, thanks!

From the logs when ran with --foreground I I do not see any conda create command.

Thanks! 🙂

Version 0.17.2 it says

Okay, but are you logs still stored on MinIO with only using sdk.development.default_output_uri ?

Yep, this install PyTorch CPU

Same error 😕

Is this really working for you guys? I have no clue what's wrong. Seems so unlikely that my code works with artifacts, datasets, but not logging...

Seems more like a bug or something is not properly configured on my side.

Tried to install cudatoolkit==11.1 manually in this environemnt and got:

Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                                         

UnsatisfiableError: The following specifications were found to be incompatible with each other:



Package xz conflicts for:
python=3....

Thu Mar 11 17:52:45 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.56       Driver Version: 460.56       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |   ...

conda env update -p .clearml/venvs-builds/3.8 ./environment.yml

with environment.yml

name: clearml
channels:
  - pytorch
  - anaconda
  - conda-forge
  - defaults
dependencies:
  - pytorch==1.8.0

But it is not related to network speed, rather to clearml. I simple file transfer test gives me approximately 1 GBit/s transfer rate between the server and the agent, which is to be expected from the 1Gbit/s network.

Agent runs in docker mode. I ran the agent on the same machine as the server this time.

I will try a minimal version now

Yea, and the script ends with clearml.Task - INFO - Waiting to finish uploads

When you say it is an SDK parameter this means that I only have to specify it on the computer where I start the task from, right? So an clearml-agent would read this parameter from the task itself.

Yea, it was finished after 20 hours. Since the artifact started uploading when the experiment finishes otherwise, there is no reporting for the the time where it uploaded. I will debug it and report what I find out

Nothing changes, still bad owner or permissions.

I guess the supported storage mediums (e.g. S3, ceph, etc...) dont have this issue, right?

ca-certificates           2021.1.19            h06a4308_1  
certifi                   2020.12.5        py38h06a4308_0  
cudatoolkit               11.0.221             h6bb024c_0  
ld_impl_linux-64          2.33.1               h53a641e_7  
libedit                   3.1.20191231         h14c3975_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.1.0                hdf63c60_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
ncurses    ...

Would it help you diagnose this problem if I ran conda env create --file=environment.yml and see whether it works?

Seems possible because I didn't know I had to specify an entrypoint somewhere. I will do some additional tests.

But this means the logger will use the default fileserver or not?

How does clearml-agent create the conda environment?

Perfect! I have to thank you for helping me, not the other way around!

Thank you for the quick reply. Maybe anyone knows whether there is an option to let docker delete images after container exit?

Thank you for helping!

The agent and server have similar hardware also. So I would expect same read/write speed.