I assume it was run using a ClearML agent?

I can see the following using docker ps:
d5330ec8c47d allegroai/clearml-agent "/usr/agent/entrypoi…" 3 weeks ago  Up 3 weeks  clearml

I execute the following to access the container
docker exec -u root -t -i clearml /bin/bash

I went to /root/.clearml/venv-builds but it is empty

which docker did you exec into?

SuccessfulKoala55 I'm currently inside the docker container to recover the ckpt files. But /root/.clearml/venvs-builds seems to be empty. Any idea where I could then find the ckpt files?

It's probably the daemon agent docker, not the one that was spun for your experiment

thanks for the info, thats really bad 😬 I thought that the output_uri defaults to the fileserver 🙄

Hi ClumsyElephant70 , by default, the agent will not send these to the server. In order to do that, you'll need to set the output_uri parameter in Task.init() (or set the agent's default_output_uri in the clearml.conf file) and point it to the ClearML file server's address

I assume you've executed the agent in docker mode?

thanks a lot, yes it was the daemon :man-facepalming: I already could recover one checkpoint!