ClearML FAQ

Answered

Hi,

Hi,
What would be the correct way to check the hash of a file before using https://clear.ml/docs/latest/docs/references/sdk/dataset/#add_files ? is there a built in function in Clear ML? What would be the correct way to add the dataframe.shape to the content/preview within clearml Dataset ?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					OutrageousSheep60
				
					0
					 × 1

Votes Newest

Answers 6

I saw https://clear.ml/docs/latest/docs/references/sdk/dataset/#verify_dataset_hash - but I don't think it is the correct one. the https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.shape.html property

  				
Posted 
	one year ago

					More
				  		
  Report
		
					OutrageousSheep60
				
					0
					 × 1

How would you report this to a Task?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

shape -> tuple([int],[int])
I decided to use

._task.upload_artifact(name='metadata', artifact_object=metadata)

where metadata is a dict

metadata = {**metadata, **{"name":f"{Path(file_tmp_path).name}", "shape": f"{df.shape}"}}

  				
Posted 
	one year ago

					More
				  		
  Report
		
					OutrageousSheep60
				
					0
					 × 1

About 2. I'm not familiar in depth with it. What does it come out as? Text?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

We need to convert it a DataFrame since

Displaying metadata in the UI is only supported for pandas Dataframes for now. Skipping!

  				
Posted 
	one year ago

					More
				  		
  Report
		
					OutrageousSheep60
				
					0
					 × 1

Hi OutrageousSheep60 ,

I guess ClearML has an internal function to check hash. You can follow the code and see what is being used and use something similar to validate.

How does you dataframe.shape look?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

978 Views

6 Answers

one year ago

Hi All, My Organisation Wants To Be Able To Trace The Changes Of All Dataset Versions From The Latest Version To The Root (Aka We Would Like To Be Able To Query And Display The Entire Version Tree). However, Clearml From My Knowledge Does Not Allow For Th

Hi There, Our. Self-Hosted Server Is Periodically Very Slow To React In The Web Ui. We'Ve Been Debugging For Quite Some Time, And It Would Seem That Elastisearch Might Be The Culprit. Looking At The Elastisearch Index, We Have An Index Of Around 80G Of Tr

Hi All, Would Someone Be Able To Help Me Understand Data Security When Working With A Local Clearml Server? I'M Working On A Project With Sensitive Data, So I'Ve Deployed A Local Clearml Server, And I Need To Confirm Whether, With The Local Server, The D

Hello! I Get This Error When Running Multiple Jobs On The Same Dataset, Would Someone Be Able To Help Debug?:

Hi, I Know That If You Have A Child Dataset Of A Dataset With Zips, And If The Parent Has Been Cached Locally, The Files In The Zips Would Be Symlinked To The Parent'S In

Hi All. I'M Trying To Start Docker Container In Agent Not As A Root. Read, That There Is Clearml_Agent_Exec_User Environment Variable For That. But Is Seems To Be Ignored, When I Pass It As With Docker'S -E Argument. What'S The Correct Way To Make Agent N

Hi All, What Options Do I Have For Setting Up Autoscaling Queues On Azure? Anything Else Other Than Kubernetes? And If Going The Kubernetes Route Is Enterprise Needed Or Would The Pro Package Be Enough? Would Scaling To 0 Be Possible? Also, Any Way To Ea

Hey Clearml Team, Would Be Nice To Include Dataset Hash When Using Clearml-Data Cli. Is The Order Of --Ids The Same As Returned Rows?

Hello. I Would Like To Ask A Question About Clearml Cloud Pricing. Where Should I Ask? I Would Like To Know The Email Address Where I Can Inquire About Clearml Pricing. The Contact Function On The Homepage Does Not Work. Third-Party Cookie Will Be Blocke