These images are actually stored there and I can access them via the url shared above (the one written in the pop up message saying that these files could not be deleted)

I got some progress TimelyPenguin76 , Now the task runs and I get the error from docker:
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

yes, that's also what I thought

Hi SuccessfulKoala55 , there it is > https://github.com/allegroai/clearml-server/issues/100

I hit F12 to check projects.get_all_ex but nothing is fired, I guess the web ui is just frozen in some weird state

Yes, that's what it looks like. Somehow when you clone the experiment repo, you correctly set the git creds in the url, but when the dependencies are installed, the git creds are not taken in account

Ok, I am asking because I often see the autoscaler starting more instances than the number of experiments in the queues, so I guess I just need to increase the max_spin_up_time_min

yes

Hi SoggyFrog26 , https://github.com/allegroai/clearml/blob/master/docs/datasets.md

` Traceback (most recent call last):
File "devops/train.py", line 73, in <module>
train(parse_args)
File "devops/train.py", line 37, in train
train_task.get_logger().set_default_upload_destination(args.artifacts + '/clearml_debug_images/')
File "/home/machine/miniconda3/envs/py36/lib/python3.6/site-packages/clearml/logger.py", line 1038, in set_default_upload_destination
uri = storage.verify_upload(folder_uri=uri)
File "/home/machine/miniconda3/envs/py36/lib/python3.6/site...

Yes, super thanks AgitatedDove14 !

Hi TimelyPenguin76 ,
trains-server: 0.16.1-320
trains: 0.15.1
trains-agent: 0.16

I made sure before deleting the old index that the number of docs matched

But clearml does read from env vars as well right? It’s not just delegating resolution to the aws cli, so it should be possible to specify the region to use for the logger, right?

I will try to isolate the bug, if I can, I will open an issue in trains-agent 🙂

I actually need to be able to overwrite files, so in my case it makes sense to give the Deleteobject permission in s3. But for other cases, why not simply catch this error, display a warning to the user and store internally that delete is not possible?

Why?

thats very cool!!

it would be nice if Task.connect_configuration could support custom yaml file readers for me

I get the same error when trying to run the task using clearml-agent services-mode with docker, so weird

Are you planning to add a server-backup service task in the near future?

I want in my CI tests to reproduce a run in an agent because the env changes and some things break in agents and not locally

Awesome! (Broken link in migration guide, step 3: https://allegro.ai/docs/deploying_trains/trains_server_es7_migration/ )

AgitatedDove14 This looks awesome! Unfortunately this would require a lot of changes in my current code, for that project I found a workaround 🙂 But I will surely use it for the next pipelines I will build!

I am using clearml_agent v1.0.0 and clearml 0.17.5 btw

AgitatedDove14 I see https://github.com/allegroai/clearml-session/blob/main/clearml_session/interactive_session_task.py#L21= that a key pair is hardcoded in the repo. Is it being used to ssh to the instance?

with open(path, "r") as stream: return yaml.load(stream, Loader=yaml.FullLoader)

AgitatedDove14 I made some progress:
In clearml.conf of the agent, I set: sdk.development.report_use_subprocess = false (because I had the feeling that Task._report_subprocess_enabled = False wasn’t taken into account) I’ve set task.set_initial_iteration(0) Now I was able to get the followin graph after resuming -

This https://discuss.elastic.co/t/index-size-explodes-after-split/150692 seems to say for the _split API such situation happens and solves itself after a couple fo days, maybe the same case for me?

SuccessfulKoala55 , This is not the exact corresponding request (I refreshed the tab since then), but the request is an events.get_task_logs , with the following content: