TimelyPenguin76 I think our problem is that the agent is not using this environment, I'm not sure which one he does... Is there a way to hard-code the agent environment?

this is the full one TimelyPenguin76

Yep, the trains server is basically a docker-compose based service.
All you have to do is change the ports in the docker-compose.yml file.

If you followed the instructions in the docs you should find that file in /opt/trains/docker-compose.yml and then you will see that there are multiple services ( apiserver , elasticsearch , redis etc.) and in each there might be a section called ports which then states the mapping of the ports.
The number on the left, is ...

AgitatedDove14 I really don't know how is this possible... I tried upgrading the server, tried whatever I could

About small toy code to reproduce I just don't have the time for that, but I will paste the callback I am using to this explanation. This is the overall logic so you can replicate and use my callback

From the pipeline task, launch some sub tasks, and put in their post_execute_callback the .collect_description_tables method from my callback class (attached below) Run t...

Okay, so let me get this straight

The autoscaling is basically an ever-running task (lets say on the services queue). Now, the actual auto scaling and which queues exist have nothign to do with that, and are configured in the auto scale task?

I'll try to put it again

Now I remind you that using the same credentials exactly, the auto scaler task could launch instances before

I'll try

https://github.com/allegroai/trains/blob/master/trains/backend_config/config.py

I might just do that, it will be my second contribution to ClearML 🙂

AgitatedDove14 sorry for delayed reply - where do I read the version the Cleanup Service is using?

I think I got it, I'll ping her again if it won't succeed

I understand it starts locally... what does it mean to run the steps locally?

later today or tomorrow, I'll update

no this is from the task execution that failed

I'll tr yto work with that

to see if it contains the .git folder

👍

Confirmed working 😄

Do you have any idea as to why does that happen SuccessfulKoala55

very cool

but I don't really know

we did execute locally

I was sure you are on Israel times as well, sorry for the night time thing 😄

And yes, it makes perfect sense, thanks for the answer

CostlyOstrich36 ?

TimelyPenguin76

and when looking at the running task, I still see the credentials

Depending on where the agent is, the value of DATA_DIR might change