ClearML FAQ | Hello I Have A Question Please, I'M Having A Hard Time Understanding How Clearml-Data Works With Docker, If I Want To Test/Train A Model Using A Dataset That I Have Locally For Example, And Then I Want To Re-Test/Train It Using Clearml-Agent Inside A Dock

Answered

Hello I Have A Question Please, I'M Having A Hard Time Understanding How Clearml-Data Works With Docker, If I Want To Test/Train A Model Using A Dataset That I Have Locally For Example, And Then I Want To Re-Test/Train It Using Clearml-Agent Inside A Dock

hello i have a question please, I'm having a hard time understanding how clearml-data works with docker, if i want to test/train a model using a dataset that I have locally for example, and then i want to re-test/train it using clearml-agent inside a docker container, how can i make it use the dataset that i have locally and not re-download a new one to use in its cache? for example for training yolov11, i get the following message even though the dataset is already available locally:
thank you in advance

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					DisturbedReindeer69
				
					0
					 × 1

Votes Newest

Answers 2

Hi @<1820993248525553664:profile|DisturbedReindeer69> , I second what @<1822805241150574592:profile|ShinySparrow39> , said. You can basically mount folders between your files system to the container using arguments such as -v , in the case of ClearML, you would want to mount the external ClearML cache folder ( ~/.clearml usually) of clearml to the internal cache folder inside the container.

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Hi,
I am not sure what you are trying to do, but docker containers do not usually have access to the files on your system. Have you tried mounting your local file using the -v docker argument at startup?

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					ShinySparrow39
				
					0
					 × 1

Write your answer

670 Views

2 Answers

4 months ago

4 months ago

Tags

Similar posts

Here'S My Question. I Have A Free Account On App.Clear.Ml And I'M Trying To Create A Dataset Consisting Of Images And Masks (Jpgs And Pngs). I Use The Following Commands:

Someone Can Help With A Problem With A Dataset Cache Download? All My Workers Are Download The Ds Every Time That My Simulation Starts. It'S Not Using The Cache And Always Download It Again. I Have Check The Config Is Fine, The Cache Dir Is Fine, The File

Hi Everyone, I'M Having An Issue With Clearml Datasets And Would Like To Know If This Is Possible. I Have A Task That Is Executed Repeatedly. This Task May Require Data To Be Loaded And Updated From A Dataset. My Question Is: Is There A Way To Append To A

Hi Everyone! I"M Building The Ml-Cluster With Shared Cephfs Over It And I Want To Spawn Several Agents On The Vm'S Do I Need To Make Separate Cache And Dataset Dirs For Each Agent? Or I Could Use Same Directory For All Cache Files? If So, Is There Some

Hi, I Am Having Troubles On Why Get_Dependency_Graph Is Not Having The Expected Behaviour. I Have The Following Test Example With A 3 Layer Dataset: When I Retrieve This From Get_Dependency_Graph I Would Expect To See Something Like The Web Interface. D

Hello, I Am Having Issues Using The Dataset Service Pointing To The Wrong Fileserver Url When Running Tasks Remotely On The Clearml Agents Running In K8S. I Was Following The Onboarding Guides Available On Youtube. In The Video They Run A Command To Uploa

Once I'Ve Created A Dataset And Finalized/Published It, I Can'T Figure Out How To Associate It With A Task Such That It Shows Up In The Task Ui. Dataset.Get() Finds A Dataset, But It Doesn'T Show Up In The Task Ui. I Can Get Inputmodels And Outputmodels T

Hey Everyone, What Is The Best Way To Export Environment Variables (Especially The Ones With Secrets Such As Db Credentials) And Make Them Available To A Clearml Agent In The Cloud? I Want To Create A Task Which Will Essentially Use A Set Of Db Credenti

Hi All, Juts Learning The Ropes Of Clearml Atm. And Am Doing A Really Simple Etl Pipeline: Raw Data -> Clean Data My Current Approach Is In One Script, I Add The Raw Data File To A Dataset In The Project: # Register_Raw.Ipynb

Hi Clearml Team, Is There Best Practice To Improve Dataset'S Storage Efficiency? For Example, I Don'T Really Need All 5 Versions Of The Same Dataset Get Saved/Remembered, Is There A Way To Prune Old Versions Of Datasets To Be More Storage Efficient?