I will try to isolate the bug, if I can, I will open an issue in trains-agent 🙂

but most likely I need to update the perms of /data as well

Ok, I could reproduce with Firefox and Chromium. Steps:
Add creds (either via the popup or in the settings) Go the /settings/webapp-configuration -> Creds should be there Hit F5 Creds are gone

There is a pinned github thread on https://github.com/allegroai/clearml/issues/81 , seems to be the right place?

This https://discuss.elastic.co/t/index-size-explodes-after-split/150692 seems to say for the _split API such situation happens and solves itself after a couple fo days, maybe the same case for me?

` Traceback (most recent call last):
File "devops/train.py", line 73, in <module>
train(parse_args)
File "devops/train.py", line 37, in train
train_task.get_logger().set_default_upload_destination(args.artifacts + '/clearml_debug_images/')
File "/home/machine/miniconda3/envs/py36/lib/python3.6/site-packages/clearml/logger.py", line 1038, in set_default_upload_destination
uri = storage.verify_upload(folder_uri=uri)
File "/home/machine/miniconda3/envs/py36/lib/python3.6/site...

Sorry both of you, my problem was actually lying somewhere else (both buckets are in the same region) - thanks for you time!

I checked the commit date anch and went to all experiments, and scrolled until finding the experiment

thanks for your help anyway AgitatedDove14 !

UnevenDolphin73 , task = clearml.Task.get_task(clearml.config.get_remote_task_id()) worked, thanks

Also enable_git_ask_pass is not dumped into the logs when an experiment start btw

thanks Jake, good to know!

Should I try to disable dynamic mapping before doing the reindex operation?

No

in clearml.conf:
agent.package_manager.system_site_packages = true agent.package_manager.pip_version = "==20.2.3"

Well not really

I can probably have a python script that checks if there are any tasks running/pending, and if not, run docker-compose down to stop the clearml-server, then use boto3 to trigger the creating of a snapshot of the EBS, then wait until it is finished, then restarts the clearml-server, wdyt?

Yes, it is supposed to run for 200 epochs

SuccessfulKoala55 Am I doing/saying something wrong regarding the problem of flushing every 5 secs (See my previous message)

SuccessfulKoala55 I want to avoid writing creds in plain text in the config file

Hi AgitatedDove14 , How should we proceed to fix this bug? Should I open an issue in github? Should I try to make a minimal reproducible example? It’s blocking me atm

It seems that around here, a Task that is created using init remotely in the main process gets its output_uri parameter ignored

(Even if I explicitely do my_task.close() )

AgitatedDove14 I finally solved it: The problem was --network='host' should be --network=host

ok, and if not the case, it will fall back to 3.8, right? Would it be possible to support such use case? (have the clearml-agent setting-up a different python version when a task needs it?)

thats very cool!!

python3 -m trains_agent --config-file "~/trains.conf" daemon --queue default --log-level DEBUG --detached --gpus 1 > ~/trains-agent.startup.log 2>&1

Hi AgitatedDove14 , so I ran 3 experiments:
One with my current implementation (using "fork") One using "forkserver" One using "forkserver" + the DataLoader optimizationI sent you the results via MP, here are the outcomes:
fork -> 101 mins, low RAM usage (5Go constant), almost no IO forkserver -> 123 mins, high RAM usage (16Go, fluctuations), high IO forkserver + DataLoader optimization: 105 mins, high RAM usage (from 28Go to 16Go), high IO
CPU/GPU curves are the same for the 3 experiments...

Yes, it did spin two instances for the same task

that would work for pytorch and clearml yes, but what about my local package?