Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Is There Way To Have Dynamic Pipeline Parameters And Use It Inside Pipeline Controller?

Is there way to have dynamic pipeline parameters and use it inside pipeline controller?

pipe.add_parameter(name="P", default="yes")
print(pipe.get_parameters())

got { 'P': 'yes'} which is ok, but if I start pipeline again from web ui with different value for P, I still get default value "yes". Any idea?

  
  
Posted 19 days ago
Votes Newest

Answers 3


Here is basic example:

from clearml import PipelineController

def step_function(param):
    print("Hello from function!")
    print("Param:", param)

if __name__ == '__main__':
    repo = '
'
    repo_branch = 'main'
    working_dir = 'pipelines'

    pipe = PipelineController(
        name='Test',
        project='Test',
        version='0.0.1',
        add_pipeline_tags=False,
        repo=repo,
        repo_branch=repo_branch,
        working_dir=working_dir
    )
    pipe.set_default_execution_queue('default')

    print("Parameters: ", pipe.get_parameters())
    pipe.add_parameter(name='pipeline_param', default='')
    print("Parameters: ", pipe.get_parameters())

    pipe.add_function_step(
        name='test_step',
        function=step_function,
        function_kwargs=dict(param='${pipeline.pipeline_param}'),
        cache_executed_step=True,
        repo=repo,
        repo_branch=repo_branch,
        working_dir=working_dir
    )

    pipe.start(queue='default')

When call python script, pipeline is created and I have right output:

# Pipeline step output:
Parameters:  {}
Parameters:  {'pipeline_param': ''}

# test_step output:
Hello from function!
Param: 

Then I started again from clearml web ui, new pipeline is created, new value for parameter is set (I can see it in web ui step info), but I got old value printed in controller.

# Pipeline step output:
Parameters:  {}
Parameters:  {'pipeline_param': ''}

# test_step output:
Hello from function!
Param: NEW_VALUE
  
  
Posted 19 days ago

@<1523701070390366208:profile|CostlyOstrich36> any idea? 🙂

  
  
Posted 19 days ago

Hi @<1702492411105644544:profile|YummyGrasshopper29> , can you provide a standalone script that reproduces this?

  
  
Posted 19 days ago
66 Views
3 Answers
19 days ago
18 days ago
Tags