Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Is There Way To Have Dynamic Pipeline Parameters And Use It Inside Pipeline Controller?

Is there way to have dynamic pipeline parameters and use it inside pipeline controller?

pipe.add_parameter(name="P", default="yes")
print(pipe.get_parameters())

got { 'P': 'yes'} which is ok, but if I start pipeline again from web ui with different value for P, I still get default value "yes". Any idea?

  
  
Posted 3 months ago
Votes Newest

Answers 3


Here is basic example:

from clearml import PipelineController

def step_function(param):
    print("Hello from function!")
    print("Param:", param)

if __name__ == '__main__':
    repo = '
'
    repo_branch = 'main'
    working_dir = 'pipelines'

    pipe = PipelineController(
        name='Test',
        project='Test',
        version='0.0.1',
        add_pipeline_tags=False,
        repo=repo,
        repo_branch=repo_branch,
        working_dir=working_dir
    )
    pipe.set_default_execution_queue('default')

    print("Parameters: ", pipe.get_parameters())
    pipe.add_parameter(name='pipeline_param', default='')
    print("Parameters: ", pipe.get_parameters())

    pipe.add_function_step(
        name='test_step',
        function=step_function,
        function_kwargs=dict(param='${pipeline.pipeline_param}'),
        cache_executed_step=True,
        repo=repo,
        repo_branch=repo_branch,
        working_dir=working_dir
    )

    pipe.start(queue='default')

When call python script, pipeline is created and I have right output:

# Pipeline step output:
Parameters:  {}
Parameters:  {'pipeline_param': ''}

# test_step output:
Hello from function!
Param: 

Then I started again from clearml web ui, new pipeline is created, new value for parameter is set (I can see it in web ui step info), but I got old value printed in controller.

# Pipeline step output:
Parameters:  {}
Parameters:  {'pipeline_param': ''}

# test_step output:
Hello from function!
Param: NEW_VALUE
  
  
Posted 3 months ago

@<1523701070390366208:profile|CostlyOstrich36> any idea? 🙂

  
  
Posted 3 months ago

Hi @<1702492411105644544:profile|YummyGrasshopper29> , can you provide a standalone script that reproduces this?

  
  
Posted 3 months ago
202 Views
3 Answers
3 months ago
3 months ago
Tags