A Question On Pipeline Artifacts And Their Use In Steps. I Want To Add An Artifact To The Pipeline (See Below), And Use It As Input To A Function Step. But The Following Does Not Work Because

Answered

A question on pipeline artifacts and their use in steps. I want to add an artifact to the pipeline (see below), and use it as input to a function step. But the following does not work because step name 'pipeline' is invalid Does anyone know if this is possible at all? Thanks for your help :)

pipe = PipelineController(name="my_pipeline", project="my_project")
pipe.upload_artifact(name="my_artifact", artifact_object=my_artifact)
pipe.add_function_step(
    name="step",
    function=step_function,
    function_kwargs=dict(param="${pipeline.artifacts.my_artifact}"),
)

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AdventurousBee56
				
					0
					 × 1

Votes Newest

Answers 3

Hi @<1649221402894536704:profile|AdventurousBee56> , I'm not sure I understand. Can you add the full log and explain step by step what's happening?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Hi @<1523701070390366208:profile|CostlyOstrich36> , here's a more full example of what I would like to be able to do:

some_object_instance = dict(a=1, b=2)

def step_func(some_object):
    print(some_object)

pipe = PipelineController(name="my_pipeline", project="my_project")
pipe.upload_artifact(name="some_object", artifact_object=some_object_instance)
pipe.add_function_step(
    name="step",
    function=step_func,
    function_kwargs=dict(param="${pipeline.some_object}"),
)
pipe.start_locally(run_pipeline_steps_locally=True)

And the error is:

"/home/xxx/site-packages/clearml/automation/controller.py", line 2177, in _add_function_step
    self.__verify_step_reference(node=self.Node(name=name), step_ref_string=v)
  File "/home/xxx/site-packages/clearml/automation/controller.py", line 3116, in __verify_step_reference
    raise ValueError("Node '{}', parameter '{}', step name '{}' is invalid".format(node.name, v, prev_step))
ValueError: Node 'step', parameter '${pipeline.some_object}', step name 'pipeline' is invalid

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AdventurousBee56
				
					0
					 × 1

I think this is just a feature that is not currently supported, but was wondering if there was a way around it?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					AdventurousBee56
				
					0
					 × 1

Write your answer

650 Views

3 Answers

10 months ago