Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Is It Possible To Add A Callback For A Pipeline From A Step?

Is it possible to add a callback for a pipeline from a step?

  
  
Posted 2 years ago
Votes Newest

Answers 30


Thanks

  
  
Posted 2 years ago

barebones, can a step in a pipeline refer to a previous step by name and get it?

  
  
Posted 2 years ago

"General" is the parameter section name (like Args)

  
  
Posted 2 years ago

Sorry, don’t get it

  
  
Posted 2 years ago

The description says this though

A section name associated with the connected object. Default: 'General'

  
  
Posted 2 years ago

Is task.parent something that could help?

Exactly 🙂 something like:
# my step is running here the_pipeline_task = Task.get_task(task_id=task.parent)

  
  
Posted 2 years ago

That is awesome!
If you feel like writing a bit about the use-case and how you solved it, I think AnxiousSeal95 will be more than happy to publish something like that 🙂

  
  
Posted 2 years ago

Yes, for datasets where we need GDPR compliance

  
  
Posted 2 years ago

From code perspective what changes?

  
  
Posted 2 years ago

Ok the doc needs fix

  
  
Posted 2 years ago

Sure, you can pass ${stage_data.id} as argument and the actual Task will get the reference step's Task ID of the current execution.
make sense ?

  
  
Posted 2 years ago

So General would have created a General instead of Args?

  
  
Posted 2 years ago

AgitatedDove14 - mean this - says name=None but text says default is General.

  
  
Posted 2 years ago

What happens if I do blah/dataset_url ?

  
  
Posted 2 years ago

See Args section in the screenshot
"Args/counter"

  
  
Posted 2 years ago

Ah ok got it.

  
  
Posted 2 years ago

Notice the pipeline step/Task at execution is not aware of the pipeline context

  
  
Posted 2 years ago

that worked, thanks AgitatedDove14

  
  
Posted 2 years ago

Ok the doc needs fix (edited)

suggestion?

  
  
Posted 2 years ago

AgitatedDove14 is it possible to get the pipeline task running a step in a step? Is task.parent something that could help?

  
  
Posted 2 years ago

In params:

parameter_override={'General/dataset_urlWhat’s the General for?

  
  
Posted 2 years ago

Think multiple hyper-paremter sections that we need to reference
(under the Tasks Configuration Tab, the Hyper parameters can have multiple sections)

  
  
Posted 2 years ago

On a related note - is it possible to get things like ${stage_data.artifacts.dataset.url} from within a task rather than passing params in add_step ?

  
  
Posted 2 years ago

I am essentially creating a EphemeralDataset abstraction and creating controlled lifecycle for it such that the data is removed after a day in experiments. Additionally and optionally, data created during a step in a pipeline can be cleared once the pipeline completes

  
  
Posted 2 years ago

So General would have created a General instead of Args?

yes,
This is a must, you have to specify the hyperparameters section you are referencing.
https://github.com/allegroai/clearml/blob/5a9155b2039413280f13dfded1121470c4c4323d/examples/pipeline/step2_data_processing.py#L21
This is actually:
task.connect(args, name='General')Basically there is no "random_state" only "General/random_state"
Make sense ?

  
  
Posted 2 years ago

Name is default None though?

  
  
Posted 2 years ago

Ephemeral Dataset, I like that! Is this like splitting a dataset for example, then training/testing, when done deleting. Making sure the entire pipeline is reproducible, but without storing the data long term?

  
  
Posted 2 years ago
490 Views
30 Answers
2 years ago
one year ago
Tags
Similar posts