Hi Folks, I Did A Deployment Of Clearml Using The K8S Helm Chart, And I Set The Agent Using K8S Glue. I Run A Task Locally, And I Went To The Ui Cloned The Experiment And Scheduled It In The Default Queue. After Doing This, I See That The Experiment Is Q

Answered

Hi folks, I did a deployment of ClearML using the K8s helm chart, and I set the agent using K8s Glue.

I run a task locally, and I went to the UI cloned the experiment and scheduled it in the default queue.
After doing this, I see that the experiment is queued in the "k8s_scheduler", and stays in a Pending state.

Any idea what might be causing the issue?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SarcasticSquirrel56
				
					0
					 × 1

Votes Newest

Answers 31

Thanks, adding environment variables to the agentservice solved it, but for the agentgroup agent, I can't see any obvious way to inject environment variables. In the helm chart template I don't see any way to pass custom environment variables to the pod

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SarcasticSquirrel56
				
					0
					 × 1

Show more results

Write your answer

4K Views

31 Answers

one year ago

one month ago

Hi All - I Am Using The Experiment Tracker And I See That Scalars View Is Not Updated After Some Time? In The Console View I See That My Training Job Terminated At Step 708000, Having Logged Scalar Metrics Every 4 Steps, However In The Scalars Pane I See

Hi All - I Am Expeiencing Some Weird Behavior Using Clearml Experiment Tracking With Hydra Configurations. My Hydra Omegaconf Configuration Object Is Not Always Being Picked Up, And I Am Unable To Consistently Reproduce It. Sometimes I Get The Omegaconf

Hi All! I Just Got Started To Learn Things Here, And I Have Some Questions. How Am I Supposed To Do To Perform An Experiment For Training A Model With The Existing Dataset On Clearml Server? So Basically What I'M Trying To Do Here Is To Get An Existing Da

Hello Everyone, I Have A Quick Question, I Am Using Clearml For An Ml Experiment Tracking Project. As Is, Clearml Is Saving A Version Of My Model After Each Epoch. Is There A Way For Clearml To Simply Save The Model Once Training Is Done And To Ignore The

Hi All, I Have A Question Regarding Clearml Task Initialization With Multithreading. I'M Using Python'S Joblib Library And The Parallel Class To Run An Experiment In Multiple Parallel Threads. The Experiment Runs To Completion But I Get Incomplete Mllogge

Hi All, I Have A Question About Model Querying. I Have Added Some Tags Manually After Uploading A Model As An Experiment To Clearml And Also Some Tags While Uploading The Model. When I Query The Model Only The Latter Are Recognized For The Search Of These

Hi, I Have A Small Question Regarding K8S Clearml-Serving Behavior. I Have In My Cluster One Gpu Of 16Gb Ram, And Another One Of 24 Gb Ram. I Have A Llm Model Fitting The 24Gb But Not The 16Gb Gpu. When I Call The Endpoint, How Will I Know To Which Gpu I

Hi Everyone, I'M Using A Self-Hosted Clearml Server And I Added A Small Nginx-Reverse Proxy To Sit In Front Of The Fileserver So At Least There'S Some Level Of Security To The Files. I'M Trying To Test It Now Using A Small Experiment, And I"M Running Into

Hi All. What Is The Most Straightforward Way To Download All Experiment-Related Data (Configs, Scalars, Debug Samples Etc.) For One Experiment From Clearml Server (Self-Hosted)? I Want To Backup Some Chosen Experiments And Also Poke Around The Experimenta