Try set docker_force_pull: true
under agent section of your agent's clearml.conf.
Answered
There Is An Annoying Bug In Docker Mode Which I Have Noticed Several Months Ago But Put Up With It For Some Reason, Despite I Kept Running Into It All The Time.
Sometimes Agents Don'T Pull Layers For Updated Docker Images If The Image With The Same Name
There is an annoying bug in Docker mode which I have noticed several months ago but put up with it for some reason, despite I kept running into it all the time.
Sometimes agents don't pull layers for updated Docker images if the image with the same name and tag (i.e. previous version) is already present there. For instance, this very often happens when I add new pip packages to an image. This leads to ImportError
s when I run the task. If I ssh to an agent and docker image rm
the image manually, then tasks pull the whole image with correct packages.
1K Views
2
Answers
2 years ago
one year ago
Tags
Similar posts