Hi, I'M Trying To Save Artifacts In A Folder Using Command

Answered

Hi,
I'm trying to save artifacts in a folder using command res= Task.current_task().upload_artifact('raw_data', artifact_object=os.path.join(save_dir))
but if the folder save_dir is a bit big (12GB), it fails to save (even when it return res = True).
It seem 12GB is a bit small of a limit when we are dealing with big data. Can we increase this limit? Is it a limit from python? clearml? server hardware?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					GrittyCormorant73
				
					0
					 × 1

Votes Newest

Answers 2

Hi GrittyCormorant73 , can you please add the error you're getting? What version of ClearML are you using?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

CostlyOstrich36 Im using clearml server 1.6. I dont get any error, that's the thing. Even the upload_artifact() function return true. But the artifact file simply does not exist on the server. The function is working fine if the artifact is 4GB, but when it is 12G, the file simply doesnt get uploaded to server.
Im using clearml server on EC2.

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					GrittyCormorant73
				
					0
					 × 1

Write your answer

1K Views

2 Answers

2 years ago

one year ago