` pipe = PipelineController(
project='examples',
name='Pipeline demo',
version='1.1',
add_pipeline_tags=False,
)
set the default execution queue to be used (per step we can override the execution)
pipe.set_default_execution_queue('default')
add pipeline components
pipe.add_parameter(
name='url',
description='url to pickle file',
default=' '
)
pipe.add_function_step(
name='step_one',
function=step_one,
function_kwargs=dict(pickle_data_url='${pipeline.url}'),
function_return=['data_frame'],
cache_executed_step=True,
) `This is in the examples
Try this, I tested it and it works:docker=pipe._parse_step_ref("${pipeline.url}")
It's hack-ish but it should work. I'll try and get a fix in one of the upcoming SDK releases that supports parsing references for parameters other than kwargs
I have another issue with pipelines, I have described it in the another thread would you mind if I tag you there? because no solution 😞
no, because I want to use pipe.add_parameter
in the docker
field of the pipe.add_function_step
not in the function_kwargs
AnxiousSeal95 Thank you so much! I will use it.
Just checking, are you just trying to use a different docker image in a task? Because then you might want to use this: https://clear.ml/docs/latest/docs/apps/clearml_task/#docker
https://clear.ml/docs/latest/docs/clearml_agent#docker-mode
Tried and as output clearml-agent
is trying to pull image '${pipeline.docker_image}'
can not convert it to the value
AHHHHHHHHHHHH! That makes more sense now 😄 😄
Checking 🙂
pipe.add_function_step(
name='step_one',
function=step_one,
function_kwargs=dict(pickle_data_url='${pipeline.url}'),
function_return=['data_frame'],
cache_executed_step=True,
#############
docker='${pipeline.url}' !!!!!! this does not work
) ``
Hey GrotesqueDog77 , so it seems like references only works on "function_kwargs" and not other function step parameter.
I'm trying to figure out if there's some workaround we can offer 🙂
function_kwargs
they work , but docker
parameter not
TimelyMouse69 the main problem is the arguments here is the code snippetpipeline = PipelineController( name="Awesome Pipeline") pipeline.add_parameter( "docker_image", default="DEFAULT_DOCKER")
And the I have functional step, where I want to use the argumentpipeline.add_function_step(name="best_step", docker="${pipeline.docker_image}"
And also I triedparameters = pipeline.get_parameters() pipeline.add_function_step(name="best_step", docker=parameters["docker_image"])
But when I execute this in the first case it passes string literal ${pipeline.docker_image}
and on the second version always default value.
But I want to use value from the arguments.