Hello :wave: ! I am trying to leverage the `retry_on_failure` with a `PipelineController` (using functions aka `add_function_step` ) to update my step parameters for the next retry. My understanding is that the step (init with `function_kwargs`) use a pic

Answered

Hello 👋 !
I am trying to leverage the retry_on_failure with a PipelineController (using functions aka add_function_step ) to update my step parameters for the next retry.
My understanding is that the step (init with function_kwargs ) use a pickle artifact from the pipeline. In the retry_on_failure I download it, update it and re-upload it at the same location but in the step console after the first fail I cannot load its param with the error:

omegaconf.errors.ValidationError: Unsupported value type: /clearml_agent_cache/storage_manager/global/046186b58393d3c12057efbdb8b7e325.train.cfg.pkl

Would that workflow be easier if I switch to PipelineDecorator or do you have any advice to make it work (I have some doubts on how I use function_kwargs )?
I attached a minimal snippet but let me know if you need more information, thanks

  				
Posted 
	one year ago

					More
				  		
  Report
		
					FunnyAlligator17
				
					0
					 × 1

Votes Newest

Answers 2

Thank you Eugen, I tested your recommandations and now it works!
Have a nice day :hugging_face:

  				
Posted 
	one year ago

					More
				  		
  Report
		
					FunnyAlligator17
				
					0
					 × 1

Hi @<1558986821491232768:profile|FunnyAlligator17> ! There are a few things you should consider:

Artifacts are not necessarily pickles. The objects you upload as artifacts can be serialized in a variety of ways. Our artifacts manager handles both serialization and deserialization. Because of this, you should not pickle the objects yourself, but specify artifact_object as being the object itself.
To get the deserialized artifact, just call task.artifacts[name].get() (not get_local_copy). This function will download the serialized object and deserialize it appropriately.

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

Write your answer

1K Views

2 Answers

one year ago