This is the issue, I will make sure wait_for_status() calls reload at the ends, so when the function returns you have the updated object

That sounds awesome! It will definitely fix my problem 🙂

In the meantime: I now do:
task.wait_for_status() task._artifacts_manager.flush() task.artifacts["output"].get()But I still get KeyError: 'output' ... Was that normal? Will it work if I replace the second line with task.refresh () ?

ubuntu18.04 is actually 64Mo, I can live with that 😛

Hi AppetizingMouse58 , I sent you the files in PM 🙂

So that I don’t loose what I worked on when stopping the session, and if I need to, I can ssh to the machine and directly access the content inside the user folder

Whohoo! Thanks 👌

(docker was install with sudo snap install docker )

using trains RC, trains-agent 0.15.0

Yes, that's what it looks like. Somehow when you clone the experiment repo, you correctly set the git creds in the url, but when the dependencies are installed, the git creds are not taken in account

And now that I restarted the server and went back into the project where I initially deleted the archived experiments, some of them are still there - I will leave them alone, too scared to do anything now 😄

Thanks TimelyPenguin76 and AgitatedDove14 ! I would like to delete artifacts/models related to the old archived experiments, but they are stored on s3. Would that be possible?

I was asking to exclude this possibility from my debugging journey 😁

in the UI the value is correct one (not empty, a string)

https://github.com/allegroai/clearml-agent.git@f019905720529acbd316bd39b67c5ab0c02fcd55 to be exact

Would be very cool if you could include this use case!

with the CLI, on a conda env located in /data

we use task.models[] 🙂

Hi SuccessfulKoala55 , thanks for the idea! the function isn’t called with atexit.register() though, maybe the way the agent kills the task is not supported by atexit

Try to spin up the instance of that type manually in that region to see if it is available

did you try with another availability zone?

Worked like a charm 👌

Sure 🙂 Opened https://github.com/allegroai/clearml/issues/568

Yes that’s what I did initially, but eventually I decided that it’s too much complexity added for nothing really, I’d rather drop omegaconf and if one day clearml supports it out of the box take advantage of it

ha wait, I removed the http:// in the host and it worked 🎉

the first problem I had, that didn’t gave useful infos, was that docker was not installed in the agent machine x)

Alright, I will try with that one

Yes AnxiousSeal95 , stopped instance meaning you don’t pay for it, but just its storage, as described https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html . So AgitatedDove14 increasing the IDLE timeout would still make me pay for the instance while they are idle.

Do you get stopped instances instantely when you ask for them?

Well that’s a good question, that’s what I observed some time ago, but according to their https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/...

I checked the server code diffs between 1.1.0 (when it was working) and 1.2.0 (when the bug appeared) and I saw many relevant changes that can introduce this bug > https://github.com/allegroai/clearml-server/compare/1.1.1...1.2.0

Super! I’ll give it a try and keep you updated here, thanks a lot for your efforts 🙏

I am happy if I can be of any help to fix that 😄

So the migration from one server to another + adding new accounts with password worked, thanks for your help!