Hey All

Answered

Hey All

Hey all . Two questions:

How do I configure clearml to use s3 as the storage backend?
How do I reload the config on a running server?

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					FoolishToad2
				
					0
					 × 1

Votes Newest

Answers 4

Hi @<1715538373919117312:profile|FoolishToad2> ,
Storage is used by the clients, to set up S3 storage, see here .
Changes to the server usually require a restart of the docker container - what exact changes are you thinking of?

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

I’m not sure yet . It’s my first day playing around with it .
As for the s3 storage, that would make the local cache only for locally run tasks, right? As in, the server stores the data on s3 and the agent will populate the local cache as needed?

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					FoolishToad2
				
					0
					 × 1

Yeah, I saw that. I didn’t see where to specify the bucket/prefix of the desired cache. It seems like this is for when the user (me) calls Storage.upload or whatever with a specific s3 Uri. I’m looking for s3 storage of all artifacts and return objects and everything.

Am I missing something?

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					FoolishToad2
				
					0
					 × 1

@<1715538373919117312:profile|FoolishToad2> , I think you're missing something. ClearML backend only holds references (links) to artifacts. Actual interaction with storage is done directly via the SDK, aka on the machine running the code

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

347 Views

4 Answers

5 months ago

Hey All, Hope You Had A Great Start To Your New Year! How Do I Change The Task Description From The Automatically Generated Task Description. I Went Through The Docs But Could Not Find What I Wanted.

Hey All, In Amazon Aws, Using The Auto Scaler, I Want To Mount My Efs On All Ec2 Workers Who Get Spawned By The Auto Scaler. Assuming I Edited My Security Group Rules, And Updated My Iam Role So It Could Access The Efs, How Do I Mount The Efs To The Machi

Hey, I'M Trying To Set Up Clearml To Work With Shared Drive Where All Models Are Going To Be Stored. Do You Know How To Store The Models During Training Simply By Saving The Path To Its Location? I Don'T Need To Send The Gigabyte-Size Files Through Clear

Hey Friends, How Do You Configure Clearml To Use An S3 Bucket? Specifically: Does

Hey, How Do I Use Local Files As Dependencies? I Have Several Tasks That I Want To Run In The Pipeline, So Optimally I Would Use Tasks From Functions. Can I Specify In Task What Local Files Do I Use That Should Be Packaged? Or Do I Have To Pack Everything

Hey, Guys, Are Any Of Your Have An Experience Of Configuring Clearml Agent To Use A Custom Non-Aws S3 Cloud Which Uses Http Endpoint. If So, How Did You Do It? I Would Really Like To Use Clearml But I Can'T Figure Out How To Set Up It Properly. I Tried Do

Hey, I Have Many Python Files. In The First Python File I Use The Following Line. Parameters = Task.Connect(Input) Now I Change The Hyperparameters On The Graphical Interface. But Now I Need The Hyperparameters In Every Python File. How Do I Have Access T

Hey I Use Allegro With Docker Mode. But I Do Not Have Access To Paths Where The Data Are(Data I Use For Training). How Can I Use "Volume Mount" With Allegro?

Hey All. Another Question - How Are Private Packages Handled/Installed So That Clearml-Agent Can Execute A Task? I Have A Bunch Of Private Repos For Communicating With The Data Warehouse. I Could Do A System-Wide Installation For It On The Clearml-Agent I