ClearML FAQ | I Am Using Pipeline From Decorators. In The Pipeline, There Is A Training Step That Returns A Model (I Want This Model To Also Be Uploaded As An Artifact On Clearml). But This Results In The Following Error:

Answered

I Am Using Pipeline From Decorators. In The Pipeline, There Is A Training Step That Returns A Model (I Want This Model To Also Be Uploaded As An Artifact On Clearml). But This Results In The Following Error:

I am using pipeline from decorators. In the pipeline, there is a training step that returns a model (I want this model to also be uploaded as an artifact on clearml). But this results in the following error:
Traceback (most recent call last):
File "/tmp/tmpl17srkeg.py", line 237, in <module>
task.upload_artifact(
File ".../.local/lib/python3.10/site-packages/clearml/task.py", line 1982, in upload_artifact
raise exception_to_raise
File ".../.local/lib/python3.10/site-packages/clearml/task.py", line 1963, in upload_artifact
if self._artifacts_manager.upload_artifact(
File ".../.local/lib/python3.10/site-packages/clearml/binding/artifacts.py", line 744, in upload_artifact
pickle.dump(artifact_object, f)
AttributeError: Can't pickle local object 'main_clf_training.<locals>.lstm_embedding_model'

Note: This occurs only when using pipelines

  				
Posted 
	one year ago

					More
				  		
  Report
		
					DilapidatedCow43
				
					0

Votes Newest

Answers 2

Hi DilapidatedCow43
I'm assuming the returned object cannot be pickled (which is ClearML's way of serializing it)
You can upload it as a model with
` uploaded_model_url = Task.current_task().update_output_model(model_path="/path/to/local/model")

...
return uploaded_model_url `wdyt?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Hi DilapidatedCow43 , which ClearML SDK version are you using?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Write your answer

515 Views

2 Answers

one year ago

one year ago

Tags

Similar posts

Hello Everybody, I Have A Quick Question, I Have Been Training Multiple Models And Have Reached A Point Where I Am Satisfied With A Model. How Do I Push My Model To S3 Server( Setup In The Clearml.Conf File)? Similar To What We Do When Pushing A Dataset

Hi Folks! I Have An Issue With Pipeline From Functions. I'M Loading And Converting Model In One Step And Trying To Pass Pytorch Model To The Next One. How Do I Deserialize My Model.Pkl Artifact Properly In The Second Step? I Have Checked The Step Code On

We Do Want To Have Control Which Files Are Logged In The Model Registry. There Is Such Option In Task.Init(), Auto_Connect_Frameworks=False Or Injecting A Dict() To It.. But Our Tasks Are Created As Being Part Of A Pipeline With Add_Function_Step(). So We

I'M Trying To Deploy The Open Source Clearml-Server, But I'M Running Into This Error: Traceback (Most Recent Call Last): File "/Usr/Local/Lib/Python3.9/Site-Packages/Pyhocon/Config_Parser.Py", Line 478, In _Resolve_Variable Return True, Config.Get(

***** Solved ***** It Appears That Clearml Hasn'T Automatically Captured The Requirements.Txt File In The Cloned Repository. Adding The Following Line Of Code At The Beginning Of My Script Fixed That:

Hi Everyone! I Use Clearml Pipelines And I Have Too Much Parameters In It So I Want To Use Configuration File. How Could I Connect Configuration File (Like `Task.Connect_Configuration_File`) But In Pipeline With Ui Interface?

Hi, I Have An Issue When Running A Pipeline Controller Remotely In Docker. Basically I Have A Module That Reads A Config File Into A Dict And Calls The Pipeline Controller, Like

Is There A Way I Can Upload An Ml Model (Raw, No Training Task) To S3 And Log It In The Models Section Of Clearml. I Succeeded With Storage Manager But Dont Know How To Make It Visible In Models Section. Also My Model Is Sometimes A Single File And Someti

Hi All! I Write A Data Processing Pipeline. It Is Necessary To Define Many Hyperparameters That Are Inconvenient To Redefine In A Pop-Up Window When Restarting The Pipeline From Ui. Is It Possible To Overrided The Parameters Through The Configuration File

Error Is Root@E0D3Dd2E3066:/App# Python3 Pipeline_From_Decorator.Py Traceback (Most Recent Call Last): File "Pipeline_From_Decorator.Py", Line 116, In <Module> Executing_Pipeline( File "/Usr/Local/Lib/Python3.8/Dist-Packages/Clearml/Automation/Con