I actually ran into the exact same problem. The agents aren't hosted on AWS though, just a in-house server.

That's wonderful! I'll have to give it a try. ;)

This turns out to be a layer-8 error . task.execute_remotely does work but there was a bug in my code and I wasn't correctly setting the reuse_task flag when run. Sorry to bother the both of you with my mistake.

Maybe the sleep between scheduler.mark_completed() and scheduler.delete() is too short? But I don't get why deleting the old scheduler task would break the new scheduler. I'm going to try testing by running the scheduler locally.

Strange, the code seems to work perfectly when I run it locally. To make it more confusing, the queue that I enqueue it to when I run it remotely is using agents from the same server that I'm running it locally from.

@<1523701435869433856:profile|SmugDolphin23> Yes. I'll try it in about 14 hours when I'm back at work and let you know how it goes. 😂

PRed: None

Is there currently a way to bind the same GPU to multiple queues? I believe the agent complains last time I tried (which was a bit ago).

That's great! I look forward to trying this out.

Hyperdatasets are the only ones that require a premium. If you're using normal datasets it should be fine.

Will this return a list of datasets?

It sounds like you didn't set up your config. Did you ever initialize clearml?

Alright, I fixed the issue with the scheduler eating itself. But now I'm still getting the same bug as two days ago. So the Scheduler process starts fine and doesn't "crash." But I don't get the config object in the web-app again. It seems to work if I run it locally.

To answer your earlier question, I'm using the app.clear.ml portal so

• WebApp: 3.20.1-1525
• Server: 3.20.1-1299
• API: 2.28
• And my Python ClearML version: 1.14

Oh, I get what's happening. That segment of the code is rerun when the task is enqueued remotely. So it's deleting itself. This also explains why it works fine locally. It's an ouroboros, the task is deleting itself.

Stability is for wimps. I live on the edge of brining down production at any moment, like a real developer. But thanks for the update! 🙃

In the debugger I can see that before starting the scheduler the test task is added:

ScheduleJob(name='Snitch-TaskScheduler', base_task_id='', base_function=<function main.<locals>.scheduler_function.<locals>.<lambda> at 0x7f05e1ab3600>, queue='services', target_project='DevOps', single_instance=False, task_parameters=None, task_overrides=None, clone_task=True, _executed_instances=None, execution_limit_hours=None, recurring=True, starting_time=datetime.datetime(2024, 1, 17, 10, 50, 28,...

@<1523701205467926528:profile|AgitatedDove14>

And the Task is still running? What's he clearml python version and webui version ?

No, the task stops (it's running remote, I haven't tested it running local).

It seems that the error is related to this part of the code block. However, when I comment this out I get the error I had 2 days ago with the missing configuration object.

Hi @<1523701205467926528:profile|AgitatedDove14> . I think I'm misunderstanding something here. I have the scheduler service running. Now that it's running how does one add a new task or remove an existing task from the scheduler? I get that I can add them before starting the scheduler service but once the service is running is there any way to connect to it and change the schedule?

I thought the advantage of this service would be we could schedule tasks just by connecting to the existing t...

I think the PR is a good idea. I read the contribution guidelines. It talks about referencing an issue. Did you want me to duplicate this issue on the repo or is it enough to link to this thread?

Sounds good. 👍

No error. Just a new task each time.

Project 2:

2024-01-22 17:21:56
task 6518c3cd13394aa4abbc8f0dc34eb763 pulled from 8a69a982f5824762aeac7b000fbf2161 by worker bigbrother:10
2024-01-22 17:22:03
Current configuration (clearml_agent v1.7.0, location: /tmp/.clearml_agent.bojpliyx.cfg):
----------------------
agent.worker_id = bigbrother:10
agent.worker_name = bigbrother
agent.force_git_ssh_protocol = true
agent.python_binary = /home/natephysics/anaconda3/bin/python
agent.package_manager.type = pip
agent.package_manager.pip_v...

Interesting approach. I'll give that a try. Thanks for the reply!

Oh, duh. I'll test that out. But I did have the agent.force_git_ssh_protocol: true

Ah, perfect.

Yes.

Will there be a recording posted?

Actually, clearing the cache on the other project might have fixed it. I just tested it out and it seems to be working.

This does appear to resolve the issue. I'll keep you updated if I find any other issues. Thanks @<1523701435869433856:profile|SmugDolphin23>