Hi Folks, I Did A Deployment Of Clearml Using The K8S Helm Chart, And I Set The Agent Using K8S Glue. I Run A Task Locally, And I Went To The Ui Cloned The Experiment And Scheduled It In The Default Queue. After Doing This, I See That The Experiment Is Q

Answered

Hi folks, I did a deployment of ClearML using the K8s helm chart, and I set the agent using K8s Glue.

I run a task locally, and I went to the UI cloned the experiment and scheduled it in the default queue.
After doing this, I see that the experiment is queued in the "k8s_scheduler", and stays in a Pending state.

Any idea what might be causing the issue?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SarcasticSquirrel56
				
					0
					 × 1

Votes Newest

Answers 31

is no agent listening to the "k8s_scheduler"

There should not be one, this is purely "virtual" , so users understand the k8s cluster is spinning their pod (sometimes it takes time, imagine EKS etc. , just visibility)

unfortunately I can't get info from the cluster

You should be able the pod in the cluster no?!
What's the Task Info panel say, can you share a screen shot ?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Show more results

Write your answer

23K Views

31 Answers

2 years ago

7 months ago

Hi! We Use Clearml Self-Hosted On A K8S Cluster. I Currently Try To Reduce The Log Level Of The Clearml Components Such As The Apiserver, Fileserver, Webserver And Clearml Serving. What I Found Is

Hi Folks. I'Ve Installed Clearml On K8S Cluster Using Helm Chart 7.11.0, If It Matters. When I Trying To Create "App Credentials" From Workspace Settings And Then Past Them To Clearml-Init - I Got The Error:

I Am Using Clearml Pro And Pretty Regularly I Will Restart An Experiment And Nothing Will Get Logged To Clearml. It Shows The Experiment Running (For Days) And It'S Running Fine On The Pc But No Scalers Or Debug Samples Are Shown. How Do We Troubleshoot T

Hi, I Am Logging Some Scalars From A Simple Ml Experiment, And Im Seing Different Curves Logged With Clearml Depending On When I Run The Code. At The Moment We Are Basically Drawing Some Random Numbers And Logging Their Values, But The Frequency Between E

Hi! I Keep Getting These Errors When Running An Ml Project Trying To Log Training Experiments. Does Anyone Know What They Mean And How I Can Fix It? Nothing Gets Logged When I Look At The Experiment On The Clearml Website.

Hey Guys, I'M A Total Newbie When It Comes To The World Of Ml, Currently Starting A Position As Devops/Mlops In A Company That Self Hosts Clearml On K8S. I Have Zero Prior Experience In The Field Of Mlops, And I'D Like To Master Clearml. It Does Look Lik

I Executed A Hyperparameter Optimization Task, But Just One Single Experiment Was Performed And Nothing. The Task Wasn'T Terminated Automatically, But Still Not Starting The Next Experiments. What'S Wrong Here? How Can I Fix This?

Hi All, I Am Trying To Add A Model To My Recently Setup K8S Self-Hosted Clearml-Serving. The Command Looks Like This And Worked Whith My Previous Docker Setup:

Hi All - I Am Using The Experiment Tracker And I See That Scalars View Is Not Updated After Some Time? In The Console View I See That My Training Job Terminated At Step 708000, Having Logged Scalar Metrics Every 4 Steps, However In The Scalars Pane I See