Hi NaughtyFish36 , which ClearML SDK and ClearML Agent versions are you using?
Yeah just checked this, the commit checks out on a different machine
error: could not write config file /root/.gitconfig: Device or resource busy Using cached repository in "/root/.clearml/vcs-cache/{repo}.git.{commit}/{repo}.git"I have noticed this, is there a reason it's using a cached repo here?
Is this commit local or was it pushed to some branch?
Can you make sure you can check out this commit on a different machine?
SuccessfulKoala55 Agent ver is 1.4.1, clearml sdk 1.7.2
Hi NaughtyFish36 ,
Execute a local training run (from within a docker container) which registers a task on our clearml serverWhen you do this, does ClearML detect the docker image that you're running on?
Initially there was the issue of no access via ssh, but that seemed to be fixed through mounting the local .ssh directory onto the docker container root. The subsequent error is the one above i.e. the reference is not a tree. However I can happily checkout that commit hash myself, yet the agent doesn't seem to be able to. Anyone any idea what's going on?
Is this commit local or was it pushed to some branch?
Just had the same issue. Your reply helped me fix it, thanks!
Seems like this was a hidden SSH key error that wasn't being revealed, it was using a cached repo rather than cloning the remote repo.
CostlyOstrich36 I use the task.set_base_docker(docker_image="some_image") to set the docker image for the task for future experiment runs, i don't think clearml detects the image i'm running on locally when registering the task