I hitted enter too fast ^^
Installing them globally via
$ pip install numpy opencv torch will install locally with warning:
Defaulting to user installation because normal site-packages is not writeable , therefore the installation will take place in ~/.local/lib/python3.6/site-packages , instead of the default one. Will this still be considered as global site-packages and still be included in experiments envs? From what I tested it does

Yes that’s correct - the weird thing is that the error shows the right detected region

Trying now your code… should take a couple of mins

SuccessfulKoala55 I want to avoid writing creds in plain text in the config file

But clearml does read from env vars as well right? It’s not just delegating resolution to the aws cli, so it should be possible to specify the region to use for the logger, right?

I am using clearml_agent v1.0.0 and clearml 0.17.5 btw

ok yes, that makes sense 👍

I’m not too fond of many user configurations, it’s confusing.

100% agree, nevertheless, how much is too many? Currently, there are only two settings in the user preferences category, so one more wouldn’t hurt?

however, clearml is open source, nothing stops you from adding the code and sending a PR

I’d be super happy to contribute yes! Nevertheless, I am not sure where to start: clearml-server repo? clearml-web repo?

CostlyOstrich36 How is clearml-session setting the ssh config?

btw shoulnd't it be CUDA_VERSION=11.0 ?

Done! Also I tried to use git cache ( https://git-scm.com/docs/git-credential-cache ) as a workaround (hoping that the first time it clones the experiment repo, it caches the creds for the next times, but I then get a different error: fatal: unable to find a suitable socket path; use --socket )

for now it just works

Sure, where can I find this file?

Stopping the server Editing the docker-compose.yml file, adding the logging section to all services Restarting the serverDocker-compose freed 10Go of logs

Yes that’s what I did initially, but eventually I decided that it’s too much complexity added for nothing really, I’d rather drop omegaconf and if one day clearml supports it out of the box take advantage of it

TimelyPenguin76 clearml 0.17.5 and clearml-agent 0.17.2

Thanks TimelyPenguin76 and AgitatedDove14 ! I would like to delete artifacts/models related to the old archived experiments, but they are stored on s3. Would that be possible?

Hi AgitatedDove14 , sorry somehow this message got lost 😄
clearml version is the latest at the time, 1.7.1 Yes, I always see the "model uploaded completed" for such stuck tasks I am using python 3.8.10

yes but they are in plain text and I would like to avoid that

How exactly is the clearml-agent killing the task?

yes

The task I cloned from is not the one I though

AgitatedDove14 I see that the default sample_frequency_per_sec=2. , but in the UI, I see that there isn’t such resolution (ie. it logs every ~120 iterations, corresponding to ~30 secs.) What is the difference with report_frequency_sec=30. ?

Nice, thanks!

I found, the filter actually has to be an iterable:
Task.get_tasks(project_name="my-project", task_name="my-task", task_filter=dict(type=["training"])))

Thanks!

The rest of the configuration is set with env variables

After I started clearml-session

might be worth documenting 😄

Hi AgitatedDove14 , thanks for the answer! I will try adding 'multiprocessing_context='forkserver' to the DataLoader. In the issue you linked, nirraviv mentionned that forkserver was slower and shared a link to another issue https://github.com/pytorch/pytorch/issues/15849#issuecomment-573921048 where someone implemented a fast variant of the DataLoader to overcome the speed problem.
Did you experiment any drop of performances using forkserver? If yes, did you test the variant suggested i...