I think it's still caching environments... I keep deleting the caches (pip, vcs, venvs-*) and running an experiment. it re-creates all these folders and even prints

Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests>=2.20.0->clearml==1.6.4->prediction-service-utilities==0.1.0) (3.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from requests>=2.20.0->clearml==1.6.4->prediction-service-utilities==0.1.0) (3.0.1)
Requirement already satisfied: sniffio>=1.1 in /root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from anyio<4,>=3.0.0->starlette==0.17.1->fastapi<0.76,>=0.73.0->prediction-service-utilities==0.1.0) (1.3.0)
Requirement already satisfied: zipp>=3.1.0 in /root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from importlib-resources>=1.4.0->jsonschema>=2.6.0->clearml==1.6.4->prediction-service-utilities==0.1.0) (3.12.1)
Adding venv into cache: /root/.clearml/venvs-builds/3.8