Hi NaughtyFish36 , which ClearML SDK and ClearML Agent versions are you using?
Just had the same issue. Your reply helped me fix it, thanks!
Is this commit local or was it pushed to some branch?
Hi NaughtyFish36 ,
Execute a local training run (from within a docker container) which registers a task on our clearml serverWhen you do this, does ClearML detect the docker image that you're running on?
Initially there was the issue of no access via ssh, but that seemed to be fixed through mounting the local .ssh directory onto the docker container root. The subsequent error is the one above i.e. the reference is not a tree. However I can happily checkout that commit hash myself, yet the agent doesn't seem to be able to. Anyone any idea what's going on?
Is this commit local or was it pushed to some branch?
Yeah just checked this, the commit checks out on a different machine
SuccessfulKoala55 Agent ver is 1.4.1, clearml sdk 1.7.2
error: could not write config file /root/.gitconfig: Device or resource busy Using cached repository in "/root/.clearml/vcs-cache/{repo}.git.{commit}/{repo}.git"I have noticed this, is there a reason it's using a cached repo here?
CostlyOstrich36 I use the task.set_base_docker(docker_image="some_image") to set the docker image for the task for future experiment runs, i don't think clearml detects the image i'm running on locally when registering the task
Can you make sure you can check out this commit on a different machine?
Seems like this was a hidden SSH key error that wasn't being revealed, it was using a cached repo rather than cloning the remote repo.