ClearML FAQ

Answered

Hi,

Hi,
What would be the correct way to check the hash of a file before using https://clear.ml/docs/latest/docs/references/sdk/dataset/#add_files ? is there a built in function in Clear ML? What would be the correct way to add the dataframe.shape to the content/preview within clearml Dataset ?

  				
Posted 
	2 years ago

					More  		
  Report
		
					OutrageousSheep60
				
					0
					 × 1

Votes Newest

Answers 6

We need to convert it a DataFrame since

Displaying metadata in the UI is only supported for pandas Dataframes for now. Skipping!

  				
Posted 
	2 years ago

					More  		
  Report
		
					OutrageousSheep60
				
					0
					 × 1

How would you report this to a Task?

  				
Posted 
	2 years ago

					More  		
  Report
		
					CostlyOstrich36
				
					0

shape -> tuple([int],[int])
I decided to use

._task.upload_artifact(name='metadata', artifact_object=metadata)

where metadata is a dict

metadata = {**metadata, **{"name":f"{Path(file_tmp_path).name}", "shape": f"{df.shape}"}}

  				
Posted 
	2 years ago

					More  		
  Report
		
					OutrageousSheep60
				
					0
					 × 1

I saw https://clear.ml/docs/latest/docs/references/sdk/dataset/#verify_dataset_hash - but I don't think it is the correct one. the https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.shape.html property

  				
Posted 
	2 years ago

					More  		
  Report
		
					OutrageousSheep60
				
					0
					 × 1

About 2. I'm not familiar in depth with it. What does it come out as? Text?

  				
Posted 
	2 years ago

					More  		
  Report
		
					CostlyOstrich36
				
					0

Hi OutrageousSheep60 ,

I guess ClearML has an internal function to check hash. You can follow the code and see what is being used and use something similar to validate.

How does you dataframe.shape look?

  				
Posted 
	2 years ago

					More  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

1K Views

6 Answers

2 years ago

Hello Everyone! I Need To Run The Pipeline On Schedule. Are There Any Restrictions On Running Pipelines Through The Scheduler? Would Hidden Project For Pipe Step Tasks Be A Problem For Scheduler?

Hello! I Was Wondering If It Would Be Possible To Integrate The Optuna Dashborard In The Same Run Of Clearml Hyperparameter Optimization. Like Passing The Sqlite Db Or Anything To Be Able To Access It Later Besides The Task Optimization Reports.

Hi All! I’M Still New To This And Would Be Very Grateful If You Could Answer My Question Or Point Me To A Topic Where This Is Discussed. I’M Working On My First Pet Project, And I Managed To Create A Dataset And Train A Model. However, After That The Mode

Hello Everyone, Quick Question Regarding The Api - What Would Be The Best Way To Trigger A New Run Of An Existing Pipeline With An Api Call?

Hello Everyone! I Am Relatively New To Clearml And To The *-Ops Concepts At All, As I Am But A Regular Python Dev. I Am Currently Trying To Implement Mlops Into Our Existing Local Infrastructure, So That We Would Be Able To Utilize Automated Data Preproc

But For Example The Pytorch Versions Would Be Different For The Two, Macbook Is Running A Different Version And My Gpu(An Nvidia Machine) An Another Version, How Can I Make Sure They Can Both Run Seamlessly Without Having Any Compatibility Issue?

Hi, What Would Be The Correct Way To Move An Existing Self-Hosted Clearml Environment (In Our Case Only Experiments. With Artifacts, Models, Debug Samples Etc.) From One Server To Another, With A Different Url And Possibly Different Port Numbers? Is Such

Hi All, I Am Working With An Instance Segmentation-Type Dataset. I Wanted To Know If There Is A Way To Visualize These Images And Their Webui In The Free Tier. I Have Seen Some Samples, But They Seem To Be Only For Hyper-Datasets. Could Some Correct Me I

Hi All, My Organisation Wants To Be Able To Trace The Changes Of All Dataset Versions From The Latest Version To The Root (Aka We Would Like To Be Able To Query And Display The Entire Version Tree). However, Clearml From My Knowledge Does Not Allow For Th