I just want to be able pass output from some step as input to some other step.
Hi VexedCat68
So if I understand correctly, the issue is this argument:parameter_override={'Args/dataset_id': '${split_dataset.split_dataset_id}', 'Args/model_id': '${get_latest_model_id.clearml_model_id}'},
I think that what is missing is telling it this an artifact:parameter_override={'Args/dataset_id': '${split_dataset.artifacts.split_dataset_id.url}', 'Args/model_id': '${get_latest_model_id.clearml_model_id}'},
You can see the example here:
https://clear.ml/docs/latest/docs/references/sdk/automation_controller_pipelinecontroller#add_step
Are you returning the "split_dataset_id" as an artifact?
These are the pipeline steps. Basically unable to pass these .
Some more of the error.
ValueError: Node train_model, parameter '${split_dataset.split_dataset_id}', input type 'split_dataset_id' is invalid
2021-12-30 16:22:00,130 - clearml.Repository Detection - WARNING - Failed auto-generating package requirements: exception SystemExit() not a BaseException subclass
Thank you. I didn't realize that the output could be accessed like this.
AgitatedDove14 Can you help me with this? Maybe something like storing the returned values or something in a variable outside the pipeline?