If You Rely On Task To Automatically Upload Artifacts, It Seems You Cannot Retrieve Them Using

Answered

If you rely on task to automatically upload artifacts, it seems you cannot retrieve them using task.artifacts - The task.artifacts is empty post upload, and when querying using Task.get_task(task_id='<SAME_ID>') the artifacts are also empty.

Only when explicitly calling task.upload_artifact(<artifact>,'artifact.pq') you can access it in the same session.

Logger shows S3 upload as expected, and the object is confirmed to exist in S3 and in ClearML GUI.

Is this intended?

clearml==1.1.1

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					TrickyRaccoon92
				
					0
					 × 1

Votes Newest

Answers 3

TrickyRaccoon92 , Hi!

Yes I believe this is the intended behavior. Since if you upload automatically you can upload many artifacts during a single run, whereas when you upload manually you create the object yourself.

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

TrickyRaccoon92 , after looking in the documentation a bit more it seems I am a bit wrong.

If you look here: https://clear.ml/docs/latest/docs/references/sdk/task/#upload_artifact

there is a parameter called wait_on_upload which is set by default to false. If you set it to 'True' then you should get the artifact object as well. This is due to the artifacts upload being asynchronous and I'm guessing the artifact didn't finish uploading by the time you called it.

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Thanks for the rapid reply

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					TrickyRaccoon92
				
					0
					 × 1

Write your answer

970 Views

3 Answers

3 years ago

one year ago