Okay, thank you for the suggestions, we'll try it out

Yeah, our internal slack bot

SOLVED: It was an expired service account key in a clearml config

This means that an agent only ever spins up one particular image? I'd like to define different container images for different tasks, possibly even build them in the process of starting a task. Is such a thing possible?

It could work but slack demands a minimum of 512x512

I will!

Thank you, I understand now :D

It is likely you have mismatched cuda. I presume you locally have cu113 but cu114 remotely. Were you running any updates lately?

This was actually a reset (of a one experiment) not a delete

Hello, a similar thing happened today. In the developer's console there was this line

https://server/api/v2.19/tasks.reset_many 504 (Gateway time-out)

No errors in logs, but that's because I restarted the deployment :(

Nothing at all. There are only 2 logs from this day, and all were at 2am

It's not because of the remote machine, it's the requirements 😅 as i said, the package is not on pypi. Try adding this at the top of your requirements.txt:

-f torch==1.12.1+cu113 ...other deps...

When installing locally you said to pip to look for packages at that page, and you dont say that to the remote pip

trigger.add_task_trigger(name='export', schedule_task_id=SCHEDULE_ID, task_overrides={...})I would like to override the commit hash of the SCHEDULE_ID with task_overrides

the entire index is 35G

clearml-agent daemon --docker --gpus all --queue Q_NAME --log-level DEBUG --detached

I guess I'll let you know the next time this happens haha

Is the trigger controller running on the services queue ?

Yes, yes it is

I haven't looked, I'll let you know next time it happens

sure

Unfortunately, no, I can't paste the whole code. In a nutshell, the trigger spawns a new GCE instance with a clearml-agent running to schedule the experiments in Cloud.
This is an excerpt:

def gcp_start_trigger(task_id: str):
    curr_task = Task.get_task(task_id)
    #curr_task.reset(force=True)
    config = extract_config(curr_task)
    machine_type = config.get('machine-type')
    queue_name = f"gcp/{machine_type}"
    ensure_queue(queue_name)  # creates a new queue if it doesn't...

Yeah, you are right.

We use an empty queue to enqueue our tasks in, just to trigger the scheduler 😅 it's only importance is that the experiment is not enqueued anywhere else, but the trigger then enqueues it

It's just that the trigger is never triggered
(Except when a new task is created - this was not the case)

agent version latest
clearml version latest

we didn't change a thing from the defaults that's in your github 😄 so it's 500M?

Yeah, sorry I typoed 😅 "newer than 18.04" was I supposed to say

What I meant was that we rebuilt them with 22.04

CostlyOstrich36 this sounds great. How do I accomplish that?

By language, I meant the syntax. What is Args and what is batch in Args/batch and what other values exist 😀

By commit hash, I mean the hash od the commit a task was run from. I wish to refer to that commit hash in another task (started with a triggerscheduler) in code

Ok great. We were writing clearml triggers and they didn't work with "aborted". 😅
I would kindly suggest perhaps adding a set of all statuses in the docs

But do consider a sort of a designer's press kit on your page haha