Hi! I'M Running Both Training And Inference Using Aws Sagemaker Pipelines, And I Want To Track Experiment Runs Using Clearml. I Was Able To Integrate Clearml At Runtime (Task.Set_Credentials And Task.Init()), And I Was Able To Track Metrics, Artifacts, E

Answered

Hi!
I'm running both training and inference using AWS SageMaker pipelines, and I want to track experiment runs using ClearML.

I was able to integrate ClearML at runtime (Task.set_credentials and Task.init()), and I was able to track metrics, artifacts, etc.
However, because it's running inside a Docker container, Git is detached.

The workaround I did was cloning the repo at runtime to get commitId, repoName, branchName, etc., but that created an overhead.

I was wondering if there is a better way.

Thanks!

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					PerplexedShells66
				
					0
					 × 1

Votes Newest

Answers 3

You can simply run the script from inside the repo once, you can also use execute_remotely to avoid actually running the entire thing

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Hi @<1835488771542355968:profile|PerplexedShells66> , I think this is what you're looking for 🙂
None

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Thanks, @<1523701070390366208:profile|CostlyOstrich36> !

Currently, I'm using Task.set_script

In both cases, I need to clone the repo, and I'm trying to avoid that.

I'm looking for an automated way to detect repo, branch, and commit it

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					PerplexedShells66
				
					0
					 × 1

Write your answer

307 Views

3 Answers

2 months ago

Hi Everyone, I'Ve Set Up Clearml With Docker Before And Was Able To See The Clearml Login Page Allowing Users To Create Accounts Via Email. When Deploying The Recommended Ami From Clearml Website I Am Unable To Find That Page Or Configure That Page. Is

Hi Team, Can Someone Help Better Understand The Prerequisites For Running A Cloned Task (If Any)? Background: I Was Able To Run And Complete A Task From Vscode. The Same Task, When Cloned And Enqueued Would Result In A File Or Directory Not Found Error. T

Dear All, Great To Join Your Community. We Are Working On Plant Growth Stage Models At Basf For Farmers And I Was Wondering If Clearml Can Be Used Also For Data Versioning Of Tabular Data, Structured Data. I Would Like To Track If This And That Row Is Par

Hi There, Maybe This Was Already Asked But I Don'T Remember: Would It Be Possible To Have The Clearml-Agent Switch Between Docker Mode And Virtualenv Mode At Runtime, Depending On The Experiment

Hi All, I Would Like To Put Table With Url Links And Image Thumnails. I Was Able To Get Url As Hyper Link However I Couldn'T Get Thumnails. Any Idea? My Code :

Hi, I Am Trying To Run Cleaml Server Locally And When Finding About Web Login Authentication I Was Not Able To Set It Up Correctly. Can You Please Tell Me The Steps To Do It And What Are The Changes Required?

Hi All, I'M Using Clearml And Pytorch-Lightning. I Was Able To Train My Models Successfully As Long As I Was Using A Single Gpu. When I Used Two Gpus For Training My Models I Got The Following Error:

Hello, I Was Wondering If Anyone Could Help With An Issue Relating To Cloned Tasks. I Have Been Able To Register And Run A Regular Task Which I Then Try To Clone Using The Examples Provided. This Runs As Expected And Gets Registered With Clearml-Server.

Hello, I Work On A Ml Project (Using Frameworks Like Pytorch And Tensorflow), I Use S3 To Save Model Checkpoints. Recently We Adopted Clearml To Track Our Experiments Which Was Indeed A Good Idea. So I Have A Small Issue With S3 And Clearml: