Hi GiganticTurtle0
I have found that
clearml
does not automatically detect the imports specified within the function decorated
The pipeline decorator will automatically detect the imports Inside the funciton, but not outside (i.e. global), to allow better control of packages (think for example one step needs the huge torch package, and the other does not.
Make sense ?
How can I tell
clearml
I will use the same virtual environment in all steps and there is no need to waste time re-installing all packages for each step?
Well, each step is a standalone (idea being, they are automatically spread across the cluster of machines, as opposed to just another subprocess),
This means each step is a standalone Task, which means "sharing" venv is not a "thing".
I am aware of the option to enable virtual environment caching, but that is still very time consuming.
venv caching makes the setup step a few seconds, do notice that it is disabled by default.
what do you mean by " that is still very time consuming" ? This is basically just copying files, it should not be more than a few seconds.
Am I missing something here ?