Thanks FiercePenguin76 , I can totally understand your point on running proper tests, and reluctance to break other things.
I suggest to add a comment with the temp fix that solved the problem for you, and we will make sure the team takes it from there. wdyt?

I thought that I can keep self._timeout_jobs = {} # Dict[datetime, str] intact and fix the part where we need to extract keys instead of values.

But the second problem hints that we need to change Dict[datetime, str] -> Dict[str, datetime] or do some custom processing before serialization

I already added to the task:
Workaround: Remove limit_execution_time from scheduler.add_task

difficult to verify correctness without a publicly available test suite

What'd you mean?

I can try, but difficult to verify correctness without a publicly available test suite

Many thanks! 😍

For me - workaround is totally acceptable, thus scheduler is once again usable for me.

Did a small update: added a workaround and renamed the issue to include more client_facing condition
limit_execution_time is presentinstead of an implementation detail condition
timeout_jobs are present

The only thing I found is that I need to run flake8, but it fails even without any changes, i.e. it was not enforced before (see my msg in )

and this can break a lot of things, when somebody start the scheduler with an older version of clearml, saves the state, then upgrades and new clearml expects the state in another format

Also added implementation thought to the issue

if a provide a PR, there I don’t see any CI processes in place that will verify the correctness of my code.

I’ll make it more visible though

FiercePenguin76 , maybe add it as PR 🙂

I think I’ll skip with PR: there is a related problem, that makes the fix (and especially its testing much more difficult): https://github.com/allegroai/clearml/issues/648#issuecomment-1102595620