ClearML FAQ | Hi Folks, I Did A Deployment Of Clearml Using The K8S Helm Chart, And I Set The Agent Using K8S Glue. I Run A Task Locally, And I Went To The Ui Cloned The Experiment And Scheduled It In The Default Queue. After Doing This, I See That The Experiment Is Q

Answered

Hi Folks, I Did A Deployment Of Clearml Using The K8S Helm Chart, And I Set The Agent Using K8S Glue. I Run A Task Locally, And I Went To The Ui Cloned The Experiment And Scheduled It In The Default Queue. After Doing This, I See That The Experiment Is Q

Hi folks, I did a deployment of ClearML using the K8s helm chart, and I set the agent using K8s Glue.

I run a task locally, and I went to the UI cloned the experiment and scheduled it in the default queue.
After doing this, I see that the experiment is queued in the "k8s_scheduler", and stays in a Pending state.

Any idea what might be causing the issue?

  				
Posted 
	2 years ago

					More  		
  Report
		
					SarcasticSquirrel56
				
					0
					 × 1

Votes Newest

Answers 31

Pretty confusing that neither

services

StickyLizard47 basically this is how a services queue agent should be spinned:
https://github.com/allegroai/clearml-server/blob/9b108740da21f25407bd2c59583ca1c86f8e1faa/docker/docker-compose.yml#L123
When spinning on a k8s cluster, this is a bit more complicated, as it needs to work with the clearml-k8s-glue.
See here how to spin it on k8s
https://github.com/allegroai/clearml-agent/tree/master/docker/k8s-glue

  				
Posted 
	2 years ago

					More  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Show more results

Write your answer

73K Views

31 Answers

2 years ago

one year ago

Tags

Similar posts

Hey Team, I Had A Question About Executing The Pipelines Using Clearml Agent Setup In K8S. When We Define A Pipeline Script To Be Executing Remotely And Submit It To The Queue For Execution We See Following Things Happening

Hi! After My Experiment Finishes, The Logs And Scalars Disappear After Some Time -- I Can See It At Task Details (Screen 1). But Scalars Are Displayed In The Usual Tabular Form (Screen 2). Additionally, All Of My Previous Experiments (For Few Years) Also

Hi Community, I Had A Clearml Experiment Seems Become "Unresponsive" Even Though It Is Showing "Running" With The Following Logging (I Didn'T Skip Any Logging Between 3-15 And 3-16, It Is Unexpected Behavior That The Experiment Just Went Mia): Any Idea Wh

Hello Everyone, I Deployed Clearml And Clearml-Agent On K8S And I Can'T Get The Agent To Use The Gpus. I Have 3 Machines With A Gpu Each Correctly Configured On The Cluster (Previously Used By

Hey, Question About The Logging Of Arguments Parsed With A Clearml Task. When I Run My Experiment And Parse An Argument Called 'Train_Dataset' With A Clearml Task Initialized, It Properly Logs The Dataset_Id To Clearml. Now The Question Is, With This Dat

Hi, I Wanted To Report A Matplotlib Figure. In The Experiment Tab Everything Works As Expected And Is Displayed Correctly. However, When Embedding This Figure In A Clearml Report, Everything Changes Colour From Black To White, But Not The Axis Labels (See

Hi There, Our Team Started Using Clearml A Few Months Ago And We'Ve Recently Deployed An Aws Eks K8S Cluster With The Hopes Of Deploying A Clearml-Agent. I'Ve Been Able To Install The Agent On The Cluster Using:

Hi! We Use Clearml Self-Hosted On A K8S Cluster. I Currently Try To Reduce The Log Level Of The Clearml Components Such As The Apiserver, Fileserver, Webserver And Clearml Serving. What I Found Is

Hi Folks. I'Ve Installed Clearml On K8S Cluster Using Helm Chart 7.11.0, If It Matters. When I Trying To Create "App Credentials" From Workspace Settings And Then Past Them To Clearml-Init - I Got The Error:

I Am Using Clearml Pro And Pretty Regularly I Will Restart An Experiment And Nothing Will Get Logged To Clearml. It Shows The Experiment Running (For Days) And It'S Running Fine On The Pc But No Scalers Or Debug Samples Are Shown. How Do We Troubleshoot T