Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Unanswered
Is There Way To Have Dynamic Pipeline Parameters And Use It Inside Pipeline Controller?


Here is basic example:

from clearml import PipelineController

def step_function(param):
    print("Hello from function!")
    print("Param:", param)

if __name__ == '__main__':
    repo = '
'
    repo_branch = 'main'
    working_dir = 'pipelines'

    pipe = PipelineController(
        name='Test',
        project='Test',
        version='0.0.1',
        add_pipeline_tags=False,
        repo=repo,
        repo_branch=repo_branch,
        working_dir=working_dir
    )
    pipe.set_default_execution_queue('default')

    print("Parameters: ", pipe.get_parameters())
    pipe.add_parameter(name='pipeline_param', default='')
    print("Parameters: ", pipe.get_parameters())

    pipe.add_function_step(
        name='test_step',
        function=step_function,
        function_kwargs=dict(param='${pipeline.pipeline_param}'),
        cache_executed_step=True,
        repo=repo,
        repo_branch=repo_branch,
        working_dir=working_dir
    )

    pipe.start(queue='default')

When call python script, pipeline is created and I have right output:

# Pipeline step output:
Parameters:  {}
Parameters:  {'pipeline_param': ''}

# test_step output:
Hello from function!
Param: 

Then I started again from clearml web ui, new pipeline is created, new value for parameter is set (I can see it in web ui step info), but I got old value printed in controller.

# Pipeline step output:
Parameters:  {}
Parameters:  {'pipeline_param': ''}

# test_step output:
Hello from function!
Param: NEW_VALUE
  
  
Posted 2 months ago
26 Views
0 Answers
2 months ago
2 months ago