Update on this one, I noticed I had different versions of clearml in my dev machine and the training machine (host and container). Updating both to the latest 1.4.1 caused a different error (related to the other question I posted in the channel) where it tries to install the packages from my dev machine (windows) in the docker container used in the training machine (ubuntu container, ubuntu host). The main issue I am trying to get around now is that I use pycocotools which has a package specific for windows ( pycocotools-windows )
This happens even though I am setting the env var CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/usr/bin/python