` from clearml import PipelineController
pipe = PipelineController(
name="Test Pipeline Controller",
project="test_pipelines",
version="1.0.0"
)
pipe.set_default_execution_queue('cpu')
pipe.add_parameter(
name='test_parameter',
description='just a random number for testing',
default='42'
)
def step_one(test_parameter):
return int(test_parameter) * 2
def step_two(new_number):
return int(new_number) * 2
pipe.add_function_step(
name='step_one',
function=step_one,
function_kwargs=dict(test_parameter='${pipeline.test_parameter}'),
function_return=['new_number'],
cache_executed_step=False,
# packages=["./requirements.txt"],
)
pipe.add_function_step(
name='step_two',
# parents=['step_one'], # the pipeline will automatically detect the dependencies based on the kwargs inputs
function=step_two,
function_kwargs=dict(new_number='${step_one.new_number}'),
function_return=['last_number'],
cache_executed_step=False,
# packages="requirements/requirements.txt",
)
pipe.start() `this is one example. I also tried other variations withs tasks and decorators, but always had the same problem. How should I specify the requirements for each step? I played with the "packages" argument, provided both paths to req files and explicit req names but the step never installed anything
I thought it might me related to this https://github.com/allegroai/clearml-agent/issues/124 but I already updated to the most recent version and it didnt help
I think it was the "force_repo_requirements_txt=true". After I changed it in some tmp file were it was loaded from I had no issues anymore
I tried them already, e.g. running https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_functions.py gives me the exact error again
Take a look at the examples here:
https://github.com/allegroai/clearml/tree/master/examples/pipeline
Hi JumpyRabbit71 , I think each step has it's own requirements