Hello everyone, I’m currently facing an issue while using Cloud ClearML with aws_autoscaler.py. Occasionally, some workers become unusable when an EC2 instance is terminated, either manually or by aws_autoscaler.py. These workers are displayed in the UI with the message “Update Time N minutes ago”. The main problem is that these workers block the entire queue, preventing the start of new tasks. When I enqueue a new task, it remains pending because the autoscaler recognizes the existing worker and doesn’t attempt to start a new EC2 instance. As a result, the only solution is to wait for the timeout of 10 minutes until the worker is removed by app.clear.ml.
Solutions I’ve considered:

1. I’ve tried removing the worker programmatically using the “workers.unregister” method. However, it only works within the same session as the workers.register. Please note that I last checked this functionality a year ago, so it might have changed since then.
2. The 10-minute interval for the timeout is not configurable and cannot be changed in app.clear.ml.
3. While I appreciate the convenience of the cloud service, I’m hesitant to deploy an on-premise version of app.clear.ml.
   If anyone knows of a workaround for this issue, please let me know. Your assistance would be greatly appreciated. Thank you.