ClearML FAQ | Hi Folks, I Just Deployed A Clearml Agent Using The Helm Chart. I Have A Few Doubts:

Answered

Hi Folks, I Just Deployed A Clearml Agent Using The Helm Chart. I Have A Few Doubts:

Hi folks, I just deployed a ClearML agent using the Helm chart. I have a few doubts:
after the deployment, I see a new queue called k8s_scheduler, which I didn't create. Is this normal? in the helm Chart, I specified that I want the AgentK8sGlue to process the queue cpu . When I enqueue an experiment to this queue, I see that there is a worker called "k8s-agent", with an ip associated, but the "currently executing" column stays empty. How can I look the logs of this agent (I can't see any deployment nor pod in the same namespace as the Agent), so I don't know where it is running. Also, I see an experiment enqueed in the "default" queue. If I delete it, it "kills" the one that was enqueued in the "cpu" queue, is this a bug?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SarcasticSquirrel56
				
					0
					 × 1

Votes Newest

Answers 31

because while I can run kubectl commands from within the agent pod, clearml doesn't seem to pick the right value:

2022-08-05 12:09:47 task 29f1645fbe1a4bb29898b1e71a8b1489 pulled from 51f5309bfb1940acb514d64931ffddb9 by worker k8s-agent-cpu 2022-08-05 12:12:59 Running kubectl encountered an error: Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2022-08-05 15:15:07 task 29f1645fbe1a4bb29898b1e71a8b1489 pulled from 51f5309bfb1940acb514d64931ffddb9 by worker k8s-agent-cpu

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SarcasticSquirrel56
				
					0
					 × 1

Show more results

Write your answer

20K Views

31 Answers

2 years ago

7 months ago

Tags

Similar posts

Hello! I Am Running Self-Hosted Clearml. Does Deleting Dataset From The Ui Also Remove It From Disk? I'Ve Just Deleted One (Send To Archive Then Delete) And Can Still See It On The Desk (

Hello Everyone! I Am New To Clearml And Currently Trying Out Its Capabilities And I Am Having One Issue With Pipelines. Pipeline Runs Properly When I Run It Locally Or If I Clone The Project And Then Run It Remotely By En-Queuing It In A Queue But If I Di

Hello Everyone. I Would Like To Ask About The Rapid Decline In Acuracy Of Opportunity Learning. When I Run The Code For Learning Locally, It Is Fine, But When I Run It In Queue, The Accuracies Suddenly Drop. No Error Is Displayed And I Want To Find Out Th

When I'M Looking At A Frames Of A Dataset Version That Is Stored In Private Blob Storage I Can'T See The Photos, It First Prompts Me To Enter A Sas Token. I Entered A Sas Token, But I Still Can'T See The Photos. I Assume This Is Because The Sas Token I En

Hi Guys, I Have A Question About Elasticsearch Connection, We Are On Kubernetes Environment (Clearml Is Deployed With Helm Chart), In The Secret We Have This :

Hi Everyone, Now This Is Happening: I Decided To Use The Default Clearml.Conf File, And I Try To Run The Agents Using Clearml-Agent Daemon --Queue Default --Docker --Foreground So, As It Is Default I Expect It To Be Able To Install Everything But Always H

Hi People When I Try To Use Docker Agents They Fail. If I Run The Command Clearml-Agent Daemon --Gpus 0 --Queue Default --Foreground Inside A Docker Prepared With All The Requirements Installed. It Tells Me:

Hi There! I Am Using A Custom Clearml Installed In K8S Using The Official Helm-Chart (With Some Modifications). I Am Trying To Set Up Training That Runs From An Engineer’S Local Laptop In The K8S Cluster Using Clearml-Task. The Single File Variant (E.G. T

Hello! Is There A Way To Filter The "Console" Logs By Host When Logging From Multiple Workers To The Same Task When Using Distributed Training? I See The "Hostname" Indicator Changes For Each Log But I Don'T See A Mechanism To Filter.

I Am Using Version 1.16.3, And There Is A New Phenomenon That When I Run A Controller With Several Tasks Or Even Just One When The Task Ends, It Is Once Again Assigned To The Queue It Was In And Is Executed Repeatedly This Phenomenon Was Not In Previous V