Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Is It Possible To Import User-Defined Modules When Wrapping Tasks/Steps With Functions And Decorators? As Far As I Know, When I Want To Define A Single “Step” In A Pipeline Using Function For Decorator, I Need To Import All Required Libs Inside This Wrapp

Is it possible to import user-defined modules when wrapping tasks/steps with functions and decorators? As far as I know, when I want to define a single “step” in a pipeline using function for decorator, I need to import all required libs inside this wrapper function. If we could only import built-in or 3rd-party libraries, these 2 ways of creating steps (function and decorator) are quite limited when comparing with using the traditional ClearML tasks.

Posted 2 years ago
Votes Newest

Answers 6

Hi GrittyKangaroo27

Is it possible to import user-defined modules when wrapping tasks/steps with functions and decorators?

Sure, any package (local included) can be imported, and will be automatically listed in the "installed packages" section of the pipeline component Task
(This of course assumes that on a remote machine you could do the "pip install <package")
Make sense ?

Posted 2 years ago

AgitatedDove14 Sorry for the confusing question. I mean I cannot use relative imports inside the “wrapping” function.

In detail, my project have this directory structure
└── project
├── package1
│ ├── build_pipeline.py
│ ├── module1.py
│ └── module2.py
└── package2
├── init.py
├── module3.py
├── module4.py
└── subpackage1
└── module5.py

From build_pipeline.py, inside each “wrapping” function, I cannot import module1and module2 of package1 and other modules in package2 as I usually do; so I think this is a limitation of building pipeline from functions and decorator.

BTW, happy weekend!

Posted 2 years ago

What do you mean by "modules first and find a way to install that package" ?
Are those modules already in wheels ? are they part a git repository?
(the pipeline component can also start inside a git repository it clones)

Posted 2 years ago

Great ascii tree 🙂
GrittyKangaroo27 assuming you are doing:
@PipelineDecorator.component(..., repo='.') def my_component(): ...The function my_component will be running in the repository root, so in thoery it could access the packages 1/2
(I'm assuming here directory "project" is the repository root)
Does that make sense ?
BTW: when you pass repo='.' to @PipelineDecorator.component it takes the current repository that exists on the local machine running the pipeline logic

Posted 2 years ago

AgitatedDove14 Nice! I’ll try this out

Posted 2 years ago

So, I have to package all the modules first and find a way to install that package at the beginning of pipeline execution to be about to use these module, am I right?

Posted 2 years ago
6 Answers
2 years ago
one year ago