Hi Max, thanks very much for your message! I understand what you’re saying now, though I suppose this is not my issue since I’m not setting any of the decorator values with variables. I’ll post a query in the main channel with code snippets to see if anyone has ideas. Thank you!
Hi ObedientDolphin41 ! Python allows you to decorate functions dynamically. See this example:
` from clearml.automation.controller import PipelineDecorator
@PipelineDecorator.component(repo=" ", repo_branch="master")
def step_one():
print('step_one')
return 1
def step_two_dynamic_decorator(repo=" ", repo_branch="master"):
@PipelineDecorator.component(repo=repo, repo_branch=repo_branch)
def step_two(arg):
print("step_two")
return arg
return step_two
@PipelineDecorator.pipeline(name='custom pipeline logic', project='examples', version='0.0.5')
def executing_pipeline():
result = step_one()
result = step_two_dynamic_decorator(repo=" ", repo_branch="2.2.2")(result)
print(result)
if name == 'main':
PipelineDecorator.run_locally()
executing_pipeline()
print('process completed') `
Reporting back: this example worked, but unfortunately did not run successfully when cloned in the UI, with an error of base_task_id is empty
akin to https://clearml.slack.com/archives/CTK20V944/p1662954750025219 previous slack thread. By editing the configuration object as mentioned above (programmatically also possible with the get and set configuration objects), the pipeline also worked when cloned 🙂
I’m also not sure but it seems like the slack trial renews from time to time in this workspace, which eventually gives access to those older threads
@<1523701225533476864:profile|ObedientDolphin41> , I was searching for anyone having an issue like me and found this thread. I have created a simple pipeline using decorators and when I try to clone it in the UI, I get that base_task_id is empty
error. It works fine when triggered programmatically from my machine. I’m wondering if you could elaborate on how you utilized the
get_configuration_object
and set_configuration_object
methods to solve this? In my case, I’m not setting a custom git repo (though I am using a custom docker image). You had linked to another thread with this issue, but unfortunately it’s hidden as it’s older than 90 days. Thank you!
Hi Mark! Do you set any of the decorator parameters using variables? That was my issue, and instead of using python variables, I hardcoded one potential value, and then used the get and set methods to change them when cloning programatically, which should be the same as changing them in the configuration tab when cloning with the UI. Hope this helps 🙂
I think I got it! I found that the branch for the component is specified in the UI in the component’s configuration object under the pipeline’s configuration tab. In theory I should be able to clone the pipeline task, use the get_configuration_object
method, change the branch, set it using the set_configuration_object
, and finally enqueue! Going to test this out
Would this then be possible by cloning the task (which is a pipeline) and accessing the right subtask (the component which should be changed)?
Awesome! Really simple and clever, love it. Thanks Eugen!
I’ve seen that you can change the branch of a cloned task like so https://github.com/allegroai/clearml-actions-train-model/blob/7f47f16b438a4b05b91537f88e8813182f39f1fe/train_model.py#L14