![Profile picture](https://clearml-web-assets.s3.amazonaws.com/scoold/avatars/ReassuredTiger98.png)
Reputation
Badges 1
606 × Eureka!Args
is similar to what is shown in print(args)
when executed remotely.
With remote_execution it is command="[...]"
, but on local it is command='train'
like it is supposed to be.
That seems to be the case. After parsing the args I run task = Task.init(...)
and then task.execute_remotely(queue_name=args.enqueue, clone=False, exit_process=True)
.
What I get for args
when I print it locally is not the same as what ClearML extracts from args
.
For now I can tell you that with conda_freeze: true
it fails, but with conda_freeze: false
it works!
So missing args that are not specified are not None
like intended, but just do not exists in args
. And command is a list instead of a single str.
Ah, it actually is also a string with remote_execution, but still not what it should be.
And in the WebUI I can see arguments similar to the second print statement's.
Latest version for everything. I will message you again, if I encounter this problem again.
I think in the paid version there is this configuration vault, so that the user can pass their own credentials securely to the agent.
Thanks a lot, now I think I understand.
Debug samples can only be controlled via api.file_server (or programatically)
Could you guide me how to approach this programmatically? Can I implement my own storage adapter for debug samples with ClearML interfaces or am I on my own?
conda env update -p .clearml/venvs-builds/3.8 ./environment.yml
with environment.yml
name: clearml
channels:
- pytorch
- anaconda
- conda-forge
- defaults
dependencies:
- pytorch==1.8.0
Setting the api.files_server:
s3://myhost:9000/clearml in clearml.conf
works!
I guess this is the current way to do it: https://github.com/tensorflow/tensorboard/issues/39#issuecomment-568917607 so I would say: Yes, it supports gif.
Thu Mar 11 17:52:45 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.56 Driver Version: 460.56 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | ...
Perfect, will try it. fyi: The conda_channels that I used are from clearml-agent init
Makes sense, but this means that we are not able to tell clearml-agent where to save on a per-task basis? I see the output_destination set correctly in clearml web interface, but as you say, clearml-agent always uses its api.fileserver
?
Or maybe a different question: What is not
Artifacts and Models. debug samples (or anything else the Logger class creates)
?
Also it is not possible to use multiple files server? E.g. log tasks on different S3 buckets without changing clearml.conf
But this seems like something that is not related to clearml 🙂 Anyways, thanks again for the explanations!
Thank you. I am not trying to use this option to speed up the setup. I have some package (carla simulator PythonAPI) that has no pip support (only easy_install). So I am thinking about just installing this manually on the worker, so that tasks can assume, that carla is provided by the system
Yea, I am still trying to get docker to work with clearml. I do not have much experience with docker besides creating Dockerfiles and it seems like the ~/.ssh/config
ownership seems broken when mounted into the container on my workstations.
Quick question: Where again does clearml place the venv? I wanna take a look into it after the task has failed
I tried to run the task with detect_with_conda_freeze: false
instead of true
and got
Executing Conda: /home/tim/miniconda3/condabin/conda install -p /home/tim/.clearml/venvs-builds/3.8 -c defaults -c conda-forge -c pytorch 'pip<20.2' --quiet --json
Pass
Conda: Trying to install requirements:
['pytorch~=1.8.0']
Executing Conda: /home/tim/miniconda3/condabin/conda env update -p /home/tim/.clearml/venvs-builds/3.8 --file /tmp/conda_envh7rq4qmc.yml --quiet --json
Conda error: Unsati...
==> 2021-03-11 13:54:59 <==
# cmd: /home/tim/miniconda3/condabin/conda create --yes --mkdir --prefix /home/tim/.clearml/venvs-builds/3.8 python=3.8
# conda version: 4.9.2
+defaults/linux-64::_libgcc_mutex-0.1-main
+defaults/linux-64::ca-certificates-2021.1.19-h06a4308_1
+defaults/linux-64::certifi-2020.12.5-py38h06a4308_0
+defaults/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
+defaults/linux-64::libedit-3.1.20191231-h14c3975_1
+defaults/linux-64::libffi-3.3-he6710b0_2
+defaults/linux-64...