Hi @<1523701435869433856:profile|SmugDolphin23> I see, but is there anyway to see the overridden config in OmegaConf so I can easily compare the difference between 2 experiments?

Hi @<1523701087100473344:profile|SuccessfulKoala55> , what preconfiguration is needed for the docker service to make? I've tried to run the docker pull manually in AWS EC2 with the same docker image without the space limit issue.

@<1523701070390366208:profile|CostlyOstrich36> Isn't the docker extra arguments only takes docker run command instead of dockerd ?

@<1523701087100473344:profile|SuccessfulKoala55> Thanks, it is the AMI issue

dockerd --storage-opt dm.basesize=20G

Hi @<1523701087100473344:profile|SuccessfulKoala55> , I just to start an EC2 instance manually and pull the docker, it is able to pull the docker without seeing the no space left issue

@<1523701070390366208:profile|CostlyOstrich36> yes, in the end of the new file

Actually never mind, it's working now!

I actually have aborted it

@<1523701205467926528:profile|AgitatedDove14> Yes I cansee the worker:

There is nothing on the queue and worker

Thanks @<1523701205467926528:profile|AgitatedDove14> . I just got an issue running clearml-task remotely, it has been working fine before today, but now every time I run clearml-task, it shows pending, and I've been waiting for 3 hours the status is still pending. The autoscalers was charging the hourly rate even though the task is still pending for 3 hours. From the console log of Clearml GPU instance, I saw it is listening to the queue, but there is no log even after 3 hours. There is not...

it has been pending whole day yesterday, but today it's able to run the task

but it still not is able to run any task after I abort and rerun another task

I got the same cuda issue after being able to use GPU

nope

It seems like CPU is working on something, I saw the usage is spiking periodically but I didn't run any task this morning

The queue will be empty when I run task

Hi @<1523701087100473344:profile|SuccessfulKoala55> I was able to solve this issue after upgrade clearml to 1.12.2, but my training/val loss become nan after the update

@<1523701087100473344:profile|SuccessfulKoala55> Hi Jake, I tried to use --output-uri in clearml-task but got the same error clearml.storage - ERROR - Failed uploading: ' LazyEval Wrapper ' object cannot be interpreted as an integer

Hi @<1523701070390366208:profile|CostlyOstrich36> , any suggestion for this error?

Hi @<1523701070390366208:profile|CostlyOstrich36> , here it is

@<1523701087100473344:profile|SuccessfulKoala55> Hi Jake, I am using 1.12.0

@<1808672071991955456:profile|CumbersomeCamel72> It can be mount without docker, but can't be mounted if I run a docker on the instance

And this issue happens randomly, I was able to run it again last night, but failed again this morning

Hi @<1523701070390366208:profile|CostlyOstrich36> , here is the configuration. The GPU could be found sometimes when I clone the previous successful run, but the GPU was found randomly. Also I am unable to run multiple task at the same time even with cloning the previous run

screenshot of AWS Autoscaler setup, cpu mode is NOT enabled

Hi @<1523701070390366208:profile|CostlyOstrich36> Any idea why this happen?

one thing I've changed is the AMI for the autoscaler, I changed it from amazon linux to ubuntu linux since my docker file size exceed the limit of the amazon linux. Not sure if this has anything to do with this issue