ClearML FAQ | I Just Deployed Clearml Into K8 Cluster Using Clearml Helm Package. When I Ran A Job, It Gave This Error In The Clearml Web Server (Attached Below). I Sshed Into The Pod Running The Clearml-Agent. Upon Typing Clearml-Agent Init, I Realised The Clearml.Con

Answered

I Just Deployed Clearml Into K8 Cluster Using Clearml Helm Package. When I Ran A Job, It Gave This Error In The Clearml Web Server (Attached Below). I Sshed Into The Pod Running The Clearml-Agent. Upon Typing Clearml-Agent Init, I Realised The Clearml.Con

I just deployed clearml into k8 cluster using clearml helm package. When i ran a job, it gave this error in the clearml web server (Attached below). I sshed into the pod running the clearml-agent. Upon typing clearml-agent init, i realised the clearml.conf is empty and also I am not able to configure it (it is write protected). I am only able to configure the clearml.conf in the agentservices node.
So i am unable to run a clearml-agent within the k8 system, however, things are working if the clearml-agent is outisde the k8 system (my laptop). Do let me know if there is some way to debug this ?

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					DeliciousBluewhale87
				
					0
					 × 1

Votes Newest

Answers 34

I did update it to clearml-agent 0.17.2 , however the issue still persists for this long-lasting service pod.
However, this issue is no more when trying to dynamically allocate pods using the Kubernetes Glue.
k8s_glue_example.py

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					DeliciousBluewhale87
				
					0
					 × 1

clearml-agent deployment file

What do you mean by that? is that the helm of the agent ?

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					DeliciousBluewhale87
				
					0
					 × 1

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					DeliciousBluewhale87
				
					0
					 × 1

Show more results

Write your answer

107K Views

34 Answers

4 years ago

one year ago

Tags

Similar posts

Hpo With Optuna Via Webapp (Pro Version) Is Not Working As Expected. I Generated A Dummy Task That Logs A Random Value Into Some Metric And Closes, The Code:

Hi Guys! Im Running Into An Issue... Im Trying To Do My Master Thesis And Clearml Is A Framework Im Investing A Lot In. And Rn Im Trying To Set Up The Serving Module Into The Server, But If I Run It On The Server I Get Connection Issues Using External Ip+

Hi Guys, Im Running Into An Issue When Creating New Clearml Dataset Version. I Want To Create New Version Of Dataset From A Local Folder, But I Don'T Want All Of The Files In The Folder To Be Included, So I Cannot Use

Within A Clearml Task, Is It Possible To Checkout Another Task Code ? Context: I Have A Bit Of A Special Use-Case: We Train Our Model Using Clearml. Our Code Prep The Data: Download Images (We Are Not Using Clearml Dataset) And Bundle Labels Into A Dataf

Hey All, I'Ve Been Doing Some Hyperparameter Tuning Today Using A Clearml Managed Dataset. However, I Kept Running Into The Following Error:

Hi, I'Ve Run Into A Problem And Would Appreciate Some Help. I Installed Clearml Locally. When I Run A New Task On A Remote Server And In The Python Training Code I Set It To Only Train On One Gpu. Everything Works Fine And I See All The Scalars Automatica

Hello Everyone! I Am Relatively New To Clearml And To The *-Ops Concepts At All, As I Am But A Regular Python Dev. I Am Currently Trying To Implement Mlops Into Our Existing Local Infrastructure, So That We Would Be Able To Utilize Automated Data Preproc

Hi All Im Trying To Save My Model Checkpoints During Runtime But Am Running Into A Confusing Snag. I'M Using The Huggingface Architecture For A Transformer. Using Their Training Module To Control Training. In The Training Args, I Have The

Hi Team, I Am Trying To Start The Clearml With Opensearch As Backend Instead Elastic Search , When I Started The Service I Am Seeing Partial Web Ui As Below Kindly Help Me In Which Module I Should Look Into It To Get Complete Functionality. No Errors I A

As Soon As I Refactor My Project Into Multiple Folders, Where On Top-Level I Put My Pipeline File, And Keep My Tasks In A Subfolder, The Clearml Agent Seems To Have Problems: