Hi Folks, I Did A Deployment Of Clearml Using The K8S Helm Chart, And I Set The Agent Using K8S Glue. I Run A Task Locally, And I Went To The Ui Cloned The Experiment And Scheduled It In The Default Queue. After Doing This, I See That The Experiment Is Q

Answered

Hi folks, I did a deployment of ClearML using the K8s helm chart, and I set the agent using K8s Glue.

I run a task locally, and I went to the UI cloned the experiment and scheduled it in the default queue.
After doing this, I see that the experiment is queued in the "k8s_scheduler", and stays in a Pending state.

Any idea what might be causing the issue?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					SarcasticSquirrel56
				
					0
					 × 1

Votes Newest

Answers 31

think it's because the proxy env var are not passed to the container ...

Yes this seems correct, the errors point to a network issues, i.e. the container does not seem to be able to connect to the clearml-server

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Show more results

Write your answer

122K Views

31 Answers

3 years ago

one year ago

Hey All, When Using Gcp Autoscaler I Always Get An Error That No Gpus Are Recognized The First Time The Autoscaler Boots Up The Vm And Runs The Code. After That First Experiment Fails I Again Run It, Autoscaler Picks Up The Same Vm And Runs The Code And

Hey All, We Have Clearml Running On Our K8S On Prem With 4 Worker Nodes For Clearml-Agents And One Node For Clearml-Server. We Would Like To Start Using Clearml Datasets And Running Pipelineg Training On It. The Datasets Might Be Very Large, About 20G. I

Anyonw Know How Can I Make Clearml Server Serving File Urls With External Domain, And Not The Internal Kubernetes Cluster Hostnames? Runnin Both The Server And Agents In K8S On-Prem, The Url Giving Is Not Reachable Because It Try To Present It As The Url

Hey All, Is There A Way To Transfer A Workspace To Someone Else? One Of My Reports Did The Original Spike On Which Experiment Tracking Platform To Use And The Company Workspace Is Under His Name. With A New Hire, We'Re Over The Seat Count Now And Blowing

Hey Team, I Had A Question About Executing The Pipelines Using Clearml Agent Setup In K8S. When We Define A Pipeline Script To Be Executing Remotely And Submit It To The Queue For Execution We See Following Things Happening

Hi! After My Experiment Finishes, The Logs And Scalars Disappear After Some Time -- I Can See It At Task Details (Screen 1). But Scalars Are Displayed In The Usual Tabular Form (Screen 2). Additionally, All Of My Previous Experiments (For Few Years) Also

Hi Community, I Had A Clearml Experiment Seems Become "Unresponsive" Even Though It Is Showing "Running" With The Following Logging (I Didn'T Skip Any Logging Between 3-15 And 3-16, It Is Unexpected Behavior That The Experiment Just Went Mia): Any Idea Wh

Hello Everyone, I Deployed Clearml And Clearml-Agent On K8S And I Can'T Get The Agent To Use The Gpus. I Have 3 Machines With A Gpu Each Correctly Configured On The Cluster (Previously Used By

Hey, Question About The Logging Of Arguments Parsed With A Clearml Task. When I Run My Experiment And Parse An Argument Called 'Train_Dataset' With A Clearml Task Initialized, It Properly Logs The Dataset_Id To Clearml. Now The Question Is, With This Dat