Hmmmmm no that's a bad solution, you're right, maybe SuccessfulKoala55 , might have an idea?
I'm just thinking If I have multiple pipelines with different purposes controllers don't have the same package requirements (some of them may take a lot of time) therefore you would need multiple files for each one. Maybe the idea of having multiple pipelines for one project is wrong - I don't actually know 🤔
I was also curious if I've missed an easy way to mention required packages for the controller, like you can do that in components (by providing packages=[...] ).
VividDucks43 , I think I might have misunderstood you a bit - For a single pipeline you would have the same requirements.txt, why would you need many?
I was also curious if I've missed an easy way to mention required packages for the controller, like you can do that in components (by providing
packages=[...]
).
I think you can do this using https://clear.ml/docs/latest/docs/references/sdk/task#taskadd_requirements
I'm not sure how the pipeline controller is handled but it still should be regarded as a task on a deeper level.
Meaning that you might need to use task = Task.current_task() and then task.add_requirements(package_name="your_package_name")
https://clear.ml/docs/latest/docs/references/sdk/task#taskcurrent_task
Hi VividDucks43 ,
I think what you're looking for is this:
https://clear.ml/docs/latest/docs/references/sdk/task#taskforce_requirements_env_freeze
🙂
So the idea of using different set of packages for pipelines in clearml is to have many different requirements.txt files?