Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Hello :wave: ! I am trying to leverage the `retry_on_failure` with a `PipelineController` (using functions aka `add_function_step` ) to update my step parameters for the next retry. My understanding is that the step (init with `function_kwargs`) use a pic

Hello 👋 !
I am trying to leverage the retry_on_failure with a PipelineController (using functions aka add_function_step ) to update my step parameters for the next retry.
My understanding is that the step (init with function_kwargs ) use a pickle artifact from the pipeline. In the retry_on_failure I download it, update it and re-upload it at the same location but in the step console after the first fail I cannot load its param with the error:

omegaconf.errors.ValidationError: Unsupported value type: /clearml_agent_cache/storage_manager/global/046186b58393d3c12057efbdb8b7e325.train.cfg.pkl

Would that workflow be easier if I switch to PipelineDecorator or do you have any advice to make it work (I have some doubts on how I use function_kwargs )?
I attached a minimal snippet but let me know if you need more information, thanks

Posted 9 months ago
Votes Newest

Answers 2

Thank you Eugen, I tested your recommandations and now it works!
Have a nice day :hugging_face:

Posted 9 months ago

Hi @<1558986821491232768:profile|FunnyAlligator17> ! There are a few things you should consider:

  • Artifacts are not necessarily pickles. The objects you upload as artifacts can be serialized in a variety of ways. Our artifacts manager handles both serialization and deserialization. Because of this, you should not pickle the objects yourself, but specify artifact_object as being the object itself.
  • To get the deserialized artifact, just call task.artifacts[name].get() (not get_local_copy). This function will download the serialized object and deserialize it appropriately.
Posted 9 months ago
2 Answers
9 months ago
9 months ago