Sure, redacted most of the params as they are sensitive:run_experiment { base_task_id = "478cfdae5ed249c18818f1c50864b83c" queue = null parents = [] timeout = null parameters { # Redacted the parameters } executed = "d1d361d1059c4f0981200f59d7683773" } segment_slides { base_task_id = "ae13cc979855482683474e9d435895bb" queue = null parents = ["run_experiment"] timeout = null parameters { Args/param = """ [ # Redacted params from here as well ['checkpoint_filename', '${run_experiment.models.output.-1.url}'], ] """ } executed = false } optimize_point_detection { base_task_id = "f91f8e36b5774cefba6aba87d85959e7" queue = null parents = ["segment_slides"] timeout = null parameters { # And here } executed = null }
Also, tried the continue_pipeline option, didn't work as it couldn't parse the previous step that run...ValueError: Could not parse reference '${run_experiment.models.output.-1.url}', step run_experiment could not be found
The pipeline stores the state of it's previous run, specifically the executed steps.
In our case the executed step was reset (I assume) so it cannot find the output model you are referring to, hence crashing
CleanPigeon16 make sense ?
CleanPigeon16 Can you send also the "Configuration Object" "Pipeline" section ?
Exception in thread Thread-5: Traceback (most recent call last): File "/opt/pyenv/versions/3.6.8/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/opt/pyenv/versions/3.6.8/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/root/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/clearml/automation/controller.py", line 615, in _daemon if self._launch_node(self._nodes[name]): File "/root/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/clearml/automation/controller.py", line 436, in _launch_node updated_hyper_parameters[k] = self._parse_step_ref(v) File "/root/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/clearml/automation/controller.py", line 787, in _parse_step_ref new_val = self.__parse_step_reference(g) File "/root/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/clearml/automation/controller.py", line 724, in __parse_step_reference step_ref_string, prev_step)) ValueError: Could not parse reference '${run_experiment.models.output.-1.url}', step run_experiment could not be found
Hi CleanPigeon16
can I make the steps in the pipeline use the latest commit in the branch?
Yes:
manually clone the stesp's Task (in the UI), and in the UI edit the Execution section and change to "last sommit on branch" and specify the branch name programmatically (as the above, clone+edit)
ValueError: Could not parse reference '${run_experiment.models.output.-1.url}', step run_experiment could not be found
Seems like the "run_experiment" step is not defined. Could that be the case?
Thanks CleanPigeon16
Could you verify Task "d1d361d1059c4f0981200f59d7683773" exists (and not archived)?
yup, it's there in draft mode so I can get the latest git commit when it's used as a base task
Is there an option to do this from a pipeline, from within the add_step method? Can you link a reference to cloning and editing a task programmatically? nope, it works well for the pipeline when not I don't choose to continue_pipeline
CleanPigeon16 Coming very soon, we adding a few features for the pipeline, this one will also be included :)
Thanks! A followup question - can I make the steps in the pipeline use the latest commit in the branch?
yup, it's there in draft mode so I can get the latest git commit when it's used as a base task
Yes that seems to be the problem, if it is in draft mode, you have no outputs...
Is there an option to do this from a pipeline, from within the
add_step
method? Can you link a reference to cloning and editing a task programmatically?
Hmm, I think there is an open GitHub issue requesting a similar ability , let me check on the progress ...
nope, it works well for the pipeline when not I don't choose to continue_pipeline
Could you send the full log please?
AgitatedDove14 is there any update on the open issue you talked about before? I think it's this one: https://github.com/allegroai/clearml/issues/214
And for some reason this clone is marked as completed. Not sure why, as it failed
Hi CleanPigeon16
Yes there is, when you are cloning the pipeline in the UI, go to the Configuration/Pipeline/continue_pipeline and change it to True