and we want to continue xecuting locally

Yeah but I don't get what it is for - for now I have 2 agents, each listening to some queues. I actually ignore the "services" queue until now

I don't get the difference between how I'm using my agents now, just starting them on machines, and making them listen to queues, to using the "services" mode

yep

you want to see its contents?

DangerousDragonfly8 but would this work if they are not concurrent but sequential?

nope - I'm talking about here

https://github.com/allegroai/trains/blob/98ea965e6d0a39019b31e5fec146ab55da0d52e8/trains/storage/manager.py#L41

I mean usually it would read if cached_file: return cached_file

TimelyPenguin76 if I build a custom image, do I have to host it on dockerhub for it to run on the agent? If not how do I make the agent aware of my custom image?

SuccessfulKoala55 The simplest thing i can think of is on Task.execute_remotely to be able to append ot the docker_init_bash_script

👍

when spinning up the ami i just went for trains recommended settings

Yes very much

but remember, it didnt work also with the default one (nvidia/cuda)

👍

essentially editing apiserver.conf section auth.fixed_users.users

I don't htink I can, this is private IP and to create a dummy example of a pipeline and execution will take me more time than I can dedicate to this

so basically - if she has new commits locally that werent pushed it won't work

But if she did not commit her latest changes, and now she enqueues - it will work?

👍

the pipeline was running

can't really see how this configuration effects the agents

or its the same palce in the config file for configuring the docker mode agent base image?

I'm trying it now

I understand it starts locally... what does it mean to run the steps locally?

I also ran it without $(pwd) on the Create Clearml task templates section, I added it because of CostlyOstrich36 's comments but it didn't help

😄

I can try do an ls to see what it will print out

I am noticing that the files are saved locally, is there any chance that the files are over-written during the run or get deleted at some point and then replaced?
Yes they are local - I don't think there is a possibility they are getting overwritten... But that depends on how clearml names them. I showed you the code that saves the artifacts, but this code runs multiple times from a given template with different values - essentially it creates like 10 times the same task with different param...

you can use pgrep -af "trains-agent"

Very nice thanks, I'm going to try the SA server + agents setup this week, let's see how it goes ✌

I guess the AMI auto updated

I might, I'll look at the internals later cause at a glance I didn't really get the logic inside get_local_copy ... the if there is ending with if ... not cached_file: return cached_file which from reading doesn't make much sense