
Reputation
Badges 1
46 × Eureka!@<1523701205467926528:profile|AgitatedDove14> I'm trying to run Clearml GPU compute(RTX 3080) with pytorch-lightning but keep getting CUDA error. Is there any specific CUDA/Ubuntu/torch/python version required? I tried several different version but can't make it work
FROM nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 as telos_algorithms
File "/code/.venv/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1013, in _run_stage
with isolate_rng():
Fi...
@<1523701070390366208:profile|CostlyOstrich36> sorry wrong log uploaded, here is the error:
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
Hi @<1523701070390366208:profile|CostlyOstrich36> , here is the configuration. The GPU could be found sometimes when I clone the previous successful run, but the GPU was found randomly. Also I am unable to run multiple task at the same time even with cloning the previous run
screenshot of AWS Autoscaler setup, cpu mode is NOT enabled
Hi @<1523701070390366208:profile|CostlyOstrich36> Any idea why this happen?
it has been pending whole day yesterday, but today it's able to run the task
Thanks @<1523701205467926528:profile|AgitatedDove14> . I just got an issue running clearml-task remotely, it has been working fine before today, but now every time I run clearml-task, it shows pending, and I've been waiting for 3 hours the status is still pending. The autoscalers was charging the hourly rate even though the task is still pending for 3 hours. From the console log of Clearml GPU instance, I saw it is listening to the queue, but there is no log even after 3 hours. There is not...
I got the same cuda issue after being able to use GPU
It seems like CPU is working on something, I saw the usage is spiking periodically but I didn't run any task this morning
Here it is @<1523701205467926528:profile|AgitatedDove14>
The queue will be empty when I run task
Actually never mind, it's working now!
Hi @<1808672071991955456:profile|CumbersomeCamel72> , the error instance is launched from ClearML AWS Autoscaler on the webpage. The sucessful mounted instance is launch manually from AWS web
@<1808672071991955456:profile|CumbersomeCamel72> It can be mount without docker, but can't be mounted if I run a docker on the instance
@<1523701070390366208:profile|CostlyOstrich36> yes, in the end of the new file
And this issue happens randomly, I was able to run it again last night, but failed again this morning
Hi @<1523701087100473344:profile|SuccessfulKoala55> , what preconfiguration is needed for the docker service to make? I've tried to run the docker pull manually in AWS EC2 with the same docker image without the space limit issue.
@<1523701205467926528:profile|AgitatedDove14> Is there any reason why you mentioned that the "correct" way to work with python and containers is to actually install everything on the system (not venv)?
There is nothing on the queue and worker
I did use --args to clearml-task command for this run, but it looks like the docker didn't take it
@<1523701205467926528:profile|AgitatedDove14> Yes I cansee the worker:
I see, seems like the -args for scripts didn't passed to the docker:
--script fluoro_motion_detection/src/run/main.py \
--args experiment=example.yaml \
but it still not is able to run any task after I abort and rerun another task
@<1523701087100473344:profile|SuccessfulKoala55> Hi Jake, I tried to use --output-uri
in clearml-task but got the same error clearml.storage - ERROR - Failed uploading: '
LazyEval Wrapper
' object cannot be interpreted as an integer
Hi @<1523701087100473344:profile|SuccessfulKoala55> I was able to solve this issue after upgrade clearml to 1.12.2, but my training/val loss become nan after the update
@<1523701087100473344:profile|SuccessfulKoala55> Hi Jake, I am using 1.12.0
Hi @<1523701070390366208:profile|CostlyOstrich36> , any suggestion for this error?