There is nothing on the queue and worker
I've added gpu:True to my hydra config file but the GPU is still not used
AgitatedDove14 Yes I cansee the worker:
It seems like CPU is working on something, I saw the usage is spiking periodically but I didn't run any task this morning
Thanks AgitatedDove14 . I just got an issue running clearml-task remotely, it has been working fine before today, but now every time I run clearml-task, it shows pending, and I've been waiting for 3 hours the status is still pending. The autoscalers was charging the hourly rate even though the task is still pending for 3 hours. From the console log of Clearml GPU instance, I saw it is listening to the queue, but there is no log even after 3 hours. There is nothing else I am running beside this one task, and seems like the worker never spin up again
2023-08-03 04:41:00,624 - clearml.Auto-Scaler - INFO - Spinning new instance resource='default', prefix='38ae71a80baf4a58893631d23c0c6e72_3090_1', queue='test-gpu'
2023-08-03 04:41:00,625 - clearml.Auto-Scaler - INFO - Creating instance for resource default
2023-08-03 04:41:01,027 - clearml.Auto-Scaler - INFO - New instance b97e702d-e2b3-4f28-adab-be59648601ea listening to test-gpu queue
And how did you connect your example,yaml?
I did use --args to clearml-task command for this run, but it looks like the docker didn't take it
EnchantingPenguin77 can you provide the full log?