Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello Everyone. I Was Following The Docs About Creating Pipelines Using Decorators (As It Looked Really Neat). Everything Worked Well For My Example, But I'Ve Noticed That Much Time For Building The Container Was Taken By Installing Packages That Weren'

Hello everyone.

I was following the docs about creating pipelines using decorators (as it looked really neat).

Everything worked well for my example, but I've noticed that much time for building the container was taken by installing packages that weren't need (albeit they are still installed and/or listed in the requirements). I wanted to know if it's possible to control the packages installed for controller of the pipeline. It's "installed packages" page contains packages that are not needed for the pipeline, while packages in components were correctly identified. It's kind of confusing for me (if at all) it is possible to set packages for the controller. I've seen a parameter for the components, but as I mentioned they were correctly identified so there is no need for my example.

I've also don't quite understand about connection of Pipeline controller to the Task. I was trying some commands like Task.ignore_requirements before calling the function which was wrapped around PipelineDecorator.pipeline but it had no effect :(

Can someone suggest how one should approach this matter? I didn't find any question about this asked previously...

  
  
Posted 2 years ago
Votes Newest

Answers 6


Hi VividDucks43 ,

I think what you're looking for is this:
https://clear.ml/docs/latest/docs/references/sdk/task#taskforce_requirements_env_freeze
🙂

  
  
Posted 2 years ago

VividDucks43 , I think I might have misunderstood you a bit - For a single pipeline you would have the same requirements.txt, why would you need many?

  
  
Posted 2 years ago

I was also curious if I've missed an easy way to mention required packages for the controller, like you can do that in components (by providing

packages=[...]

).

I think you can do this using https://clear.ml/docs/latest/docs/references/sdk/task#taskadd_requirements
I'm not sure how the pipeline controller is handled but it still should be regarded as a task on a deeper level.

Meaning that you might need to use task = Task.current_task() and then task.add_requirements(package_name="your_package_name")
https://clear.ml/docs/latest/docs/references/sdk/task#taskcurrent_task

  
  
Posted 2 years ago

I'm just thinking If I have multiple pipelines with different purposes controllers don't have the same package requirements (some of them may take a lot of time) therefore you would need multiple files for each one. Maybe the idea of having multiple pipelines for one project is wrong - I don't actually know 🤔

I was also curious if I've missed an easy way to mention required packages for the controller, like you can do that in components (by providing packages=[...] ).

  
  
Posted 2 years ago

Hmmmmm no that's a bad solution, you're right, maybe SuccessfulKoala55 , might have an idea?

  
  
Posted 2 years ago

So the idea of using different set of packages for pipelines in clearml is to have many different requirements.txt files?

  
  
Posted 2 years ago
895 Views
6 Answers
2 years ago
one year ago
Tags