ClearML FAQ | Hi Folks, I Did A Deployment Of Clearml Using The K8S Helm Chart, And I Set The Agent Using K8S Glue. I Run A Task Locally, And I Went To The Ui Cloned The Experiment And Scheduled It In The Default Queue. After Doing This, I See That The Experiment Is Q

Answered

Hi Folks, I Did A Deployment Of Clearml Using The K8S Helm Chart, And I Set The Agent Using K8S Glue. I Run A Task Locally, And I Went To The Ui Cloned The Experiment And Scheduled It In The Default Queue. After Doing This, I See That The Experiment Is Q

Hi folks, I did a deployment of ClearML using the K8s helm chart, and I set the agent using K8s Glue.

I run a task locally, and I went to the UI cloned the experiment and scheduled it in the default queue.
After doing this, I see that the experiment is queued in the "k8s_scheduler", and stays in a Pending state.

Any idea what might be causing the issue?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SarcasticSquirrel56
				
					0
					 × 1

Votes Newest

Answers 31

Martin I told you I can't access the resources in the cluster unfortunately

😞
so it seems there is some misconfiguration of the k8s glue, because we can see it can "talk" to the clearml-server, but it seems it fails to actually create the k8s pod/job. I would start with debugging the k8s glue (not the services agents). Regardless, I think the next step is to get a log of the k8s glue pod, and better understand the issue.
wdyt?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Show more results

Write your answer

23K Views

31 Answers

2 years ago

7 months ago

Tags

Similar posts

Hi There, Our Team Started Using Clearml A Few Months Ago And We'Ve Recently Deployed An Aws Eks K8S Cluster With The Hopes Of Deploying A Clearml-Agent. I'Ve Been Able To Install The Agent On The Cluster Using:

Hi! We Use Clearml Self-Hosted On A K8S Cluster. I Currently Try To Reduce The Log Level Of The Clearml Components Such As The Apiserver, Fileserver, Webserver And Clearml Serving. What I Found Is

Hi Folks. I'Ve Installed Clearml On K8S Cluster Using Helm Chart 7.11.0, If It Matters. When I Trying To Create "App Credentials" From Workspace Settings And Then Past Them To Clearml-Init - I Got The Error:

I Am Using Clearml Pro And Pretty Regularly I Will Restart An Experiment And Nothing Will Get Logged To Clearml. It Shows The Experiment Running (For Days) And It'S Running Fine On The Pc But No Scalers Or Debug Samples Are Shown. How Do We Troubleshoot T

Hi, I Am Logging Some Scalars From A Simple Ml Experiment, And Im Seing Different Curves Logged With Clearml Depending On When I Run The Code. At The Moment We Are Basically Drawing Some Random Numbers And Logging Their Values, But The Frequency Between E

Hi! I Keep Getting These Errors When Running An Ml Project Trying To Log Training Experiments. Does Anyone Know What They Mean And How I Can Fix It? Nothing Gets Logged When I Look At The Experiment On The Clearml Website.

Hey Guys, I'M A Total Newbie When It Comes To The World Of Ml, Currently Starting A Position As Devops/Mlops In A Company That Self Hosts Clearml On K8S. I Have Zero Prior Experience In The Field Of Mlops, And I'D Like To Master Clearml. It Does Look Lik

I Executed A Hyperparameter Optimization Task, But Just One Single Experiment Was Performed And Nothing. The Task Wasn'T Terminated Automatically, But Still Not Starting The Next Experiments. What'S Wrong Here? How Can I Fix This?

Hi All, I Am Trying To Add A Model To My Recently Setup K8S Self-Hosted Clearml-Serving. The Command Looks Like This And Worked Whith My Previous Docker Setup:

Hi All - I Am Using The Experiment Tracker And I See That Scalars View Is Not Updated After Some Time? In The Console View I See That My Training Job Terminated At Step 708000, Having Logged Scalar Metrics Every 4 Steps, However In The Scalars Pane I See