Thanks

I manually deleted the allegroai/trains:latest image, that didn't help either

Very nice thanks, I'm going to try the SA server + agents setup this week, let's see how it goes ✌

Committing that notebook with changes solved it, but I wonder why it failed

Mmm maybe, lets see if I get this straight

A static artifact is a one-upload object, a dynamic artifact is an object I can change during the experiment -> this results at the end of an experiment in an object to be saved under a given name regardless if it was dynamic or not?

nope - I'm talking about here

https://github.com/allegroai/trains/blob/98ea965e6d0a39019b31e5fec146ab55da0d52e8/trains/storage/manager.py#L41

😄

You should try trains-agent daemon --gpus device=0,1 --queue dual_gpu --docker --foreground and if it doesn't work try quoting trains-agent daemon --gpus '"device=0,1"' --queue dual_gpu --docker --foreground

Okay SuccessfulKoala55 , problem solved! Indeed the problem was that there is not .git folder. I updated necessary things to make the checkout action get the actual repo and now it works

It wasn't really clear to me what "standalone" means, maybe it will be better to add to the error

Error: Standalone

(no .git folder found)

script detected 'tasks/hp_optimization.py', but no requirements provided

Okay so regarding the version - we are using 1.1.1

The thing with this error it that it happens sometimes, and when it happens it never goes away...
I don't know what causes it, but we have one host where it works okay, then someone else checks out the repo and tried and it fails for this error, while another guy can do the same and it will work for him

Worth mentioning, nothing has changed before we executed this, it worked before and now after the update it breaks

I assume trains passes it as is, so I think the quoting I mentioned might work

Okay so that is a bit complicated

In our setup, the DSes don't really care about agents, the agents are being managed by our MLops team.
So essentially if you imagine it the use case looks like that:
A data scientists wants to execute some CPU heavy task. The MLops team supplied him with a queue name, and the data scientist knows that when he needs something heavy he pushes it there - the DS doesn't know nothing about where it is executed, the execution environment is fully managed by the ML...

thx TimelyPenguin76
skimming over this, I can't find how to filter by project name or something similar

I only found Project ID, which I'm not sure what this refers to - I have the project name

I did, but I only find task name, I'm looking to filter by the project

Hahahah thanks for the help SuccessfulKoala55 & CostlyOstrich36

I really do feel it would be a nice to have the ability to easily configure the Cleanup Service to cleanup only specific projects / tasks as its a common use case to have a project dedicated for debugging and alike

bottom line I want to edit the cleanup service code to only delete tasks under a specific project - how do I do that?

Example code? I didn't see anywhere an example of filtering using project name

Can I force it to output the static image?

I'll tr yto work with that

Does that mean that teh AWS autoscaler in trains, manages EC2 auto scaling directly without using the AWS built in EC2 auto scaler?

e.g. docker-compose restart

When I ran the clearml-task --name ... -project ... -script .... it failed saying not requiremetns was found

I'm quite confused... The package is not missing, it is in my environment and executing tasks normally ( python my_script.py.... ) works

I mean if I continue and build on the example in the docs, what will happen if the training task is completed, and then I get it and log to it? Will it be defined as running again?

It works!