Unanswered
Is There Way To Have Dynamic Pipeline Parameters And Use It Inside Pipeline Controller?
Here is basic example:
from clearml import PipelineController
def step_function(param):
print("Hello from function!")
print("Param:", param)
if __name__ == '__main__':
repo = '
'
repo_branch = 'main'
working_dir = 'pipelines'
pipe = PipelineController(
name='Test',
project='Test',
version='0.0.1',
add_pipeline_tags=False,
repo=repo,
repo_branch=repo_branch,
working_dir=working_dir
)
pipe.set_default_execution_queue('default')
print("Parameters: ", pipe.get_parameters())
pipe.add_parameter(name='pipeline_param', default='')
print("Parameters: ", pipe.get_parameters())
pipe.add_function_step(
name='test_step',
function=step_function,
function_kwargs=dict(param='${pipeline.pipeline_param}'),
cache_executed_step=True,
repo=repo,
repo_branch=repo_branch,
working_dir=working_dir
)
pipe.start(queue='default')
When call python script, pipeline is created and I have right output:
# Pipeline step output:
Parameters: {}
Parameters: {'pipeline_param': ''}
# test_step output:
Hello from function!
Param:
Then I started again from clearml web ui, new pipeline is created, new value for parameter is set (I can see it in web ui step info), but I got old value printed in controller.
# Pipeline step output:
Parameters: {}
Parameters: {'pipeline_param': ''}
# test_step output:
Hello from function!
Param: NEW_VALUE
20 Views
0
Answers
2 months ago
2 months ago