It intermittently reads the requirements.txt between my repo and the cache
Can you please elaborate?
Do you have resource monitoring on that machine? Any chance that that something ran out of space or memory or cpu?
What's the version of your ClearML-Agent?
Are all agents running on the same machine or is it spread out?
CostlyOstrich36 hmmm i doubt so, i'm the only one using the machine for this particular experiment at the moment.
Well it sounds like it makes some sense. Try the following on the machine running the agent - In ~/clearml.conf
edit the following section:agent.vcs_cache.enabled=False
CostlyOstrich36 I'm using clearml_agent v1.1.2 on multiple agents in the same machine
CostlyOstrich36 ahhh i suspect the error might be coming from using a cached repository? e.g. Using cached repository in "/root/.clearml/vcs-cache/<my repository> it seems like it is trying to install the requirements.txt that was cached but isnt available anymore and there are occasions where the installed packages do not reflect a complete list of what was specified in the repository's requirements.txt. Could this be a possibility for the error (either not detecting the complete list of packages or an entire cached requirements.txt file)?
It intermittently reads the requirements.txt between my repo and the cache. Im wondering if there is anyway to circumvent the cache?