Most of the time is took by building wheels for nympy and pandas which are apparently deps of clearml-agent if I read the log correctly

Well I think most of the time is took by the setup of the venv installing the packages defined in the imports in the pipeline component which is normal and some of those package have a wheel that takes a long time to build but most of those packages where already included on the Docker image I provided and I get that message in my logs:

:: Python virtual environment cache is disabled. To accelerate spin-up time setagent.venvs_cache.path=~/.clearml/venvs-cache:::

It takes about 30 seconds here for that step

sure
I hope that network incident is not too big 🙂

I think it's inside the container since it's after the worker pulls the image

Oh that makes more sense, I mean it should not build the from source, but make sense
To solve for build for source:
Add to the "Additional ClearML Configuration" section the following line:
agent.package_manager.pip_version: "<21"
You can also turn on venv caching
Add to the "Additional ClearML Configuration" section the following line:
agent.venvs_cache.path: ~/.clearml/venvs-cache
I will make sure we bump the minimum pip by default so this does not happen

It takes 20mins to build the venv environment needed by the clearml-agent

You are Joking?! 😭
it does apt-get install python3-pip , and pip install clearml-agent, how is that 20min?

Hi FierceHamster54

Could you provide an example and we will try to reproduce it?

FierceHamster54 what you are saying that Inside the container it took 20 min to run? or that spinning the GCP instance until it registered as an Agent took 20min ?

Most of the time is took by building wheels for

nympy

and

pandas

...

BTW: This happens if there is a version mismatch and pip decides it needs to build the numpy from source, Can you send the full logs of that? Maybe we can somehow avoid that?

or in another environment?

Thanks for your reactivity 🎉

AgitatedDove14 Here you go, I think it's inside the container since it's after the worker pulls the image

Does it take the same amount of time if you build it locally?

Well having a network inbcidient at HQ so this doesn't help.... but I'll keep you updqted with the tests I run tommorow