Interesting approach. I'll give that a try. Thanks for the reply!

Let me give that a try. Thanks for all the help.

None

What version of ClearML server are you using?

Looks good 😄

This doesn't really make a lot of sense. ClearML would be better served for tracking which version of the code you used for a corresponding task and you'd use something like github or gitlab to track code and host your code. You could use ClearML to help you reconstruct the environment and code from a task given it's being tracked by git and hosted somewhere you can access.

Ah, perfect.

If I wanted to do this with the ID, how would I approach it?

Ah, I think I see the issue. In my head I was crossing ID with URL.

In this case it's the ID of the "output" model from the first task.

I will add a gh issue. Is this part open source? Could I make a PR?

In the mean time I still need to implement this with the current version of ClearML. So the only way would be to have one variable per parent? Is there any smarter way to work around it?

On the original model.

After some digging we found it was actually caused by the routers IPS protection. I thought it would be strange for github to be throttling things at this scale.

Since this could happen with a lot of services, maybe it would be worth a retry option? Especially if it's part of a pipeline.

Hi @<1523701205467926528:profile|AgitatedDove14> . I think I'm misunderstanding something here. I have the scheduler service running. Now that it's running how does one add a new task or remove an existing task from the scheduler? I get that I can add them before starting the scheduler service but once the service is running is there any way to connect to it and change the schedule?

I thought the advantage of this service would be we could schedule tasks just by connecting to the existing t...

Well, if I stop the cron service and start it back up I don't have to re-register each schedule. If, for instance, I start the TaskScheduler, register a task, and stop the scheduler, how do I restart the TaskScheduler in a way that re-register the tasks? Because, in theory, they could be registered from several users and I might be unaware of tasks that were previously scheduled. What is the best practices to preserve state?

I made a video of the Scheduler config error. You can see that the same code run locally works and doesn't on remote. (I just uploaded the video so the quality might suffer until YT finishes processing the higher resolution versions).

None

It seems that the error is related to this part of the code block. However, when I comment this out I get the error I had 2 days ago with the missing configuration object.

Stability is for wimps. I live on the edge of brining down production at any moment, like a real developer. But thanks for the update! 🙃

Alright, I fixed the issue with the scheduler eating itself. But now I'm still getting the same bug as two days ago. So the Scheduler process starts fine and doesn't "crash." But I don't get the config object in the web-app again. It seems to work if I run it locally.

To answer your earlier question, I'm using the app.clear.ml portal so

• WebApp: 3.20.1-1525
• Server: 3.20.1-1299
• API: 2.28
• And my Python ClearML version: 1.14

Thanks so much @<1523701205467926528:profile|AgitatedDove14> !

@<1523701205467926528:profile|AgitatedDove14>

And the Task is still running? What's he clearml python version and webui version ?

No, the task stops (it's running remote, I haven't tested it running local).

That's what I was getting at. It wasn't clear to me from the documentation that it saves the state.

Thanks again for the info. I might experiment with it to see first hand what the advantages are.

@<1523701205467926528:profile|AgitatedDove14> Then it isn't working at intended. To test it I started the scheduler and set a simple dead man snitch process to run once a day. In the web-app (on your site app.cleearml.ml), when looking at the scheduler process in the DevOps section, I was able to see a configuration file under artifacts but it was not as all obvious how you'd change that because it wasn't part of the configuration section, it was just an artifact. So I thought maybe it was b...

Maybe the sleep between scheduler.mark_completed() and scheduler.delete() is too short? But I don't get why deleting the old scheduler task would break the new scheduler. I'm going to try testing by running the scheduler locally.

✨ It works ✨

Thanks @<1523701205467926528:profile|AgitatedDove14> 😁

Yes, I'm experimenting with this. I actually wrote my own process to do this so I just had to adapt it as a callable to pass to the scheduler. However, I'm running into an issue and I don't think this is a user error this time. When I start the scheduler, it starts running, shows up in the web-app, but then an error message in the web-app pops up Fetch parents failed and the Scheduler task disappears from the web-app. I can't even see an error log because the task is gone.

I'm running th...

In the debugger I can see that before starting the scheduler the test task is added:

ScheduleJob(name='Snitch-TaskScheduler', base_task_id='', base_function=<function main.<locals>.scheduler_function.<locals>.<lambda> at 0x7f05e1ab3600>, queue='services', target_project='DevOps', single_instance=False, task_parameters=None, task_overrides=None, clone_task=True, _executed_instances=None, execution_limit_hours=None, recurring=True, starting_time=datetime.datetime(2024, 1, 17, 10, 50, 28,...

Results:

I first tried uncommenting enable_git_ask_pass: false but it didn't resolve the issue.

I then cleared the cache in the vcs-cache folder, and that did fix the issue. This is the second time the cache seemed to have been the root cause of the problem. At some point I did move from token-based auth to ssh keys. Would this require clearing the cache for any project that was cached prior to the auth change?