Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All! Question About Pipelines Using Decorators. The First Step Of My Pipeline Uses A Specific Repo, Specified Using

Hi all! Question about pipelines using decorators.
The first step of my pipeline uses a specific repo, specified using PipelineDecorator.component(repo='repo', repo_branch='branch') .
Is there a way to dynamically change the branch of this repo?
Use case: when someone creates a branch for this repo and changes the code, we would like to run a CI/CD script that clones the latest version of this pipeline and adjusts the branch of the component, to see the impact on the pipeline’s output metrics

  
  
Posted one year ago
Votes Newest

Answers 10


Awesome! Really simple and clever, love it. Thanks Eugen!

  
  
Posted one year ago

Reporting back: this example worked, but unfortunately did not run successfully when cloned in the UI, with an error of base_task_id is empty akin to https://clearml.slack.com/archives/CTK20V944/p1662954750025219 previous slack thread. By editing the configuration object as mentioned above (programmatically also possible with the get and set configuration objects), the pipeline also worked when cloned 🙂

  
  
Posted one year ago

Hi ObedientDolphin41 ! Python allows you to decorate functions dynamically. See this example:
` from clearml.automation.controller import PipelineDecorator

@PipelineDecorator.component(repo=" ", repo_branch="master")
def step_one():
print('step_one')
return 1

def step_two_dynamic_decorator(repo=" ", repo_branch="master"):
@PipelineDecorator.component(repo=repo, repo_branch=repo_branch)
def step_two(arg):
print("step_two")
return arg
return step_two

@PipelineDecorator.pipeline(name='custom pipeline logic', project='examples', version='0.0.5')
def executing_pipeline():
result = step_one()
result = step_two_dynamic_decorator(repo=" ", repo_branch="2.2.2")(result)
print(result)

if name == 'main':
PipelineDecorator.run_locally()
executing_pipeline()
print('process completed') `

  
  
Posted one year ago

Would this then be possible by cloning the task (which is a pipeline) and accessing the right subtask (the component which should be changed)?

  
  
Posted one year ago

I’m also not sure but it seems like the slack trial renews from time to time in this workspace, which eventually gives access to those older threads

  
  
Posted one year ago

I think I got it! I found that the branch for the component is specified in the UI in the component’s configuration object under the pipeline’s configuration tab. In theory I should be able to clone the pipeline task, use the get_configuration_object method, change the branch, set it using the set_configuration_object , and finally enqueue! Going to test this out

  
  
Posted one year ago

Hi Mark! Do you set any of the decorator parameters using variables? That was my issue, and instead of using python variables, I hardcoded one potential value, and then used the get and set methods to change them when cloning programatically, which should be the same as changing them in the configuration tab when cloning with the UI. Hope this helps 🙂

  
  
Posted one year ago

Hi Max, thanks very much for your message! I understand what you’re saying now, though I suppose this is not my issue since I’m not setting any of the decorator values with variables. I’ll post a query in the main channel with code snippets to see if anyone has ideas. Thank you!

  
  
Posted one year ago

@<1523701225533476864:profile|ObedientDolphin41> , I was searching for anyone having an issue like me and found this thread. I have created a simple pipeline using decorators and when I try to clone it in the UI, I get that base_task_id is empty error. It works fine when triggered programmatically from my machine. I’m wondering if you could elaborate on how you utilized the get_configuration_object and set_configuration_object methods to solve this? In my case, I’m not setting a custom git repo (though I am using a custom docker image). You had linked to another thread with this issue, but unfortunately it’s hidden as it’s older than 90 days. Thank you!

  
  
Posted one year ago