withÂ
PipelineController
, is there any way to avoid creating a new development environment for each step of the pipeline?
You are in luck, we are expanding the PipelineController to support functions. basically allowing you to run the step on the node running the entire pipeline, but I'm not sure this covers all angles of the problem.
My main question here is, who/how the initial setup is created by cleaml-agent ?
I would like to be more efficient and re-use that environment once configured in the first task.
You have full venv caching. which means the second time a node creates the same env it will reuse the previous one. By default this is turned off because the storage requirements for the node might increase (copy of the entire pythin env might be a few GB).
un-comment this like to activate it.
https://github.com/allegroai/clearml-agent/blob/aede6f4bac71c8fc56e7cf982318a48527953a3c/docs/clearml.conf#L104