Hi Folks, I Did A Deployment Of Clearml Using The K8S Helm Chart, And I Set The Agent Using K8S Glue. I Run A Task Locally, And I Went To The Ui Cloned The Experiment And Scheduled It In The Default Queue. After Doing This, I See That The Experiment Is Q

Answered

Hi folks, I did a deployment of ClearML using the K8s helm chart, and I set the agent using K8s Glue.

I run a task locally, and I went to the UI cloned the experiment and scheduled it in the default queue.
After doing this, I see that the experiment is queued in the "k8s_scheduler", and stays in a Pending state.

Any idea what might be causing the issue?

  				
Posted 
	2 years ago

					More  		
  Report
		
					SarcasticSquirrel56
				
					0
					 × 1

Votes Newest

Answers 31

I can see the outputs from argo, so I know if some resource has been created but I can't inspect the full logs,
the ones I have available are all records similar to
No tasks in queue 80247f703053470fa60718b4dff7a576

  				
Posted 
	2 years ago

					More  		
  Report
		
					SarcasticSquirrel56
				
					0
					 × 1

Show more results

Write your answer

69K Views

31 Answers

2 years ago

12 months ago

Hi Community, I Had A Clearml Experiment Seems Become "Unresponsive" Even Though It Is Showing "Running" With The Following Logging (I Didn'T Skip Any Logging Between 3-15 And 3-16, It Is Unexpected Behavior That The Experiment Just Went Mia): Any Idea Wh

Hello Everyone, I Deployed Clearml And Clearml-Agent On K8S And I Can'T Get The Agent To Use The Gpus. I Have 3 Machines With A Gpu Each Correctly Configured On The Cluster (Previously Used By

Hey, Question About The Logging Of Arguments Parsed With A Clearml Task. When I Run My Experiment And Parse An Argument Called 'Train_Dataset' With A Clearml Task Initialized, It Properly Logs The Dataset_Id To Clearml. Now The Question Is, With This Dat

Hi, I Wanted To Report A Matplotlib Figure. In The Experiment Tab Everything Works As Expected And Is Displayed Correctly. However, When Embedding This Figure In A Clearml Report, Everything Changes Colour From Black To White, But Not The Axis Labels (See

Hi There, Our Team Started Using Clearml A Few Months Ago And We'Ve Recently Deployed An Aws Eks K8S Cluster With The Hopes Of Deploying A Clearml-Agent. I'Ve Been Able To Install The Agent On The Cluster Using:

Hi! We Use Clearml Self-Hosted On A K8S Cluster. I Currently Try To Reduce The Log Level Of The Clearml Components Such As The Apiserver, Fileserver, Webserver And Clearml Serving. What I Found Is

Hi Folks. I'Ve Installed Clearml On K8S Cluster Using Helm Chart 7.11.0, If It Matters. When I Trying To Create "App Credentials" From Workspace Settings And Then Past Them To Clearml-Init - I Got The Error:

I Am Using Clearml Pro And Pretty Regularly I Will Restart An Experiment And Nothing Will Get Logged To Clearml. It Shows The Experiment Running (For Days) And It'S Running Fine On The Pc But No Scalers Or Debug Samples Are Shown. How Do We Troubleshoot T

Hi, I Am Logging Some Scalars From A Simple Ml Experiment, And Im Seing Different Curves Logged With Clearml Depending On When I Run The Code. At The Moment We Are Basically Drawing Some Random Numbers And Logging Their Values, But The Frequency Between E