Well I think most of the time is took by the setup of the venv installing the packages defined in the imports in the pipeline component which is normal and some of those package have a wheel that takes a long time to build but most of those packages where already included on the Docker image I provided and I get that message in my logs:
:: Python virtual environment cache is disabled. To accelerate spin-up time setagent.venvs_cache.path=~/.clearml/venvs-cache:::
It takes about 30 seconds here for that step
Does it take the same amount of time if you build it locally?
AgitatedDove14 Here you go, I think it's inside the container since it's after the worker pulls the image
It takes 20mins to build the venv environment needed by the clearml-agent
You are Joking?! ðŸ˜
it does apt-get install python3-pip , and pip install clearml-agent, how is that 20min?
I think it's inside the container since it's after the worker pulls the image
Oh that makes more sense, I mean it should not build the from source, but make sense
To solve for build for source:
Add to the "Additional ClearML Configuration" section the following line:agent.package_manager.pip_version: "<21"
You can also turn on venv caching
Add to the "Additional ClearML Configuration" section the following line:agent.venvs_cache.path: ~/.clearml/venvs-cache
I will make sure we bump the minimum pip by default so this does not happen
Hi FierceHamster54
Could you provide an example and we will try to reproduce it?
Most of the time is took by building wheels for nympy and pandas which are apparently deps of clearml-agent if I read the log correctly
Well having a network inbcidient at HQ so this doesn't help.... but I'll keep you updqted with the tests I run tommorow
FierceHamster54 what you are saying that Inside the container it took 20 min to run? or that spinning the GCP instance until it registered as an Agent took 20min ?
Most of the time is took by building wheels for
nympy
and
pandas
...
BTW: This happens if there is a version mismatch and pip decides it needs to build the numpy from source, Can you send the full logs of that? Maybe we can somehow avoid that?
sure
I hope that network incident is not too big 🙂