ClearML FAQ | I Am Using Pipeline From Decorators. In The Pipeline, There Is A Training Step That Returns A Model (I Want This Model To Also Be Uploaded As An Artifact On Clearml). But This Results In The Following Error:

Answered

I Am Using Pipeline From Decorators. In The Pipeline, There Is A Training Step That Returns A Model (I Want This Model To Also Be Uploaded As An Artifact On Clearml). But This Results In The Following Error:

I am using pipeline from decorators. In the pipeline, there is a training step that returns a model (I want this model to also be uploaded as an artifact on clearml). But this results in the following error:
Traceback (most recent call last):
File "/tmp/tmpl17srkeg.py", line 237, in <module>
task.upload_artifact(
File ".../.local/lib/python3.10/site-packages/clearml/task.py", line 1982, in upload_artifact
raise exception_to_raise
File ".../.local/lib/python3.10/site-packages/clearml/task.py", line 1963, in upload_artifact
if self._artifacts_manager.upload_artifact(
File ".../.local/lib/python3.10/site-packages/clearml/binding/artifacts.py", line 744, in upload_artifact
pickle.dump(artifact_object, f)
AttributeError: Can't pickle local object 'main_clf_training.<locals>.lstm_embedding_model'

Note: This occurs only when using pipelines

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					DilapidatedCow43
				
					0

Votes Newest

Answers 2

Hi DilapidatedCow43
I'm assuming the returned object cannot be pickled (which is ClearML's way of serializing it)
You can upload it as a model with
` uploaded_model_url = Task.current_task().update_output_model(model_path="/path/to/local/model")

...
return uploaded_model_url `wdyt?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Hi DilapidatedCow43 , which ClearML SDK version are you using?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Write your answer

972 Views

2 Answers

2 years ago

one year ago

Tags

Similar posts

As Soon As I Refactor My Project Into Multiple Folders, Where On Top-Level I Put My Pipeline File, And Keep My Tasks In A Subfolder, The Clearml Agent Seems To Have Problems:

Hello Everyone, I Have An Video Classification Model. I Have A Lot Of Metadata About The Videos In A Csv File, I First Preprocess The Metadata And In Pandas Df. (There May Be Lost Videos, Invalid Videos Etc, So I Remove Those From The Df). I Want To Log T

Hello! I Can'T Seem To Be Able To Stop Clearml From Automatically Logging Model Files (Optimizer, Scheduler). It'S A Useful Feature But I'D Like To Have Some Control Over It, So That The Disk Space In My File Storage Isn'T Overused. I'M Using

Hello Everybody, I Have A Quick Question, I Have Been Training Multiple Models And Have Reached A Point Where I Am Satisfied With A Model. How Do I Push My Model To S3 Server( Setup In The Clearml.Conf File)? Similar To What We Do When Pushing A Dataset

Hi Folks! I Have An Issue With Pipeline From Functions. I'M Loading And Converting Model In One Step And Trying To Pass Pytorch Model To The Next One. How Do I Deserialize My Model.Pkl Artifact Properly In The Second Step? I Have Checked The Step Code On

We Do Want To Have Control Which Files Are Logged In The Model Registry. There Is Such Option In Task.Init(), Auto_Connect_Frameworks=False Or Injecting A Dict() To It.. But Our Tasks Are Created As Being Part Of A Pipeline With Add_Function_Step(). So We

I'M Trying To Deploy The Open Source Clearml-Server, But I'M Running Into This Error: Traceback (Most Recent Call Last): File "/Usr/Local/Lib/Python3.9/Site-Packages/Pyhocon/Config_Parser.Py", Line 478, In _Resolve_Variable Return True, Config.Get(

***** Solved ***** It Appears That Clearml Hasn'T Automatically Captured The Requirements.Txt File In The Cloned Repository. Adding The Following Line Of Code At The Beginning Of My Script Fixed That:

Hi Everyone! I Use Clearml Pipelines And I Have Too Much Parameters In It So I Want To Use Configuration File. How Could I Connect Configuration File (Like `Task.Connect_Configuration_File`) But In Pipeline With Ui Interface?

Hi, I Have An Issue When Running A Pipeline Controller Remotely In Docker. Basically I Have A Module That Reads A Config File Into A Dict And Calls The Pipeline Controller, Like