ClearML FAQ | I Just Deployed Clearml Into K8 Cluster Using Clearml Helm Package. When I Ran A Job, It Gave This Error In The Clearml Web Server (Attached Below). I Sshed Into The Pod Running The Clearml-Agent. Upon Typing Clearml-Agent Init, I Realised The Clearml.Con

Answered

I Just Deployed Clearml Into K8 Cluster Using Clearml Helm Package. When I Ran A Job, It Gave This Error In The Clearml Web Server (Attached Below). I Sshed Into The Pod Running The Clearml-Agent. Upon Typing Clearml-Agent Init, I Realised The Clearml.Con

I just deployed clearml into k8 cluster using clearml helm package. When i ran a job, it gave this error in the clearml web server (Attached below). I sshed into the pod running the clearml-agent. Upon typing clearml-agent init, i realised the clearml.conf is empty and also I am not able to configure it (it is write protected). I am only able to configure the clearml.conf in the agentservices node.
So i am unable to run a clearml-agent within the k8 system, however, things are working if the clearml-agent is outisde the k8 system (my laptop). Do let me know if there is some way to debug this ?

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					DeliciousBluewhale87
				
					0
					 × 1

Votes Newest

Answers 34

Ohh okay something seems to half work in terms of configuration, the agent has enough configuration to register itself, but fails to pass it to the task.
Can you test with the latest agent RC:
0.17.2rc4

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

By the way, are you editing the values directly? Why not use the values file?

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					FriendlySquid61
				
					0

Hi AgitatedDove14 , I also fiddled around by changing this line and restarted the deployment. But this just causes it revert back 0.17.2rc4 again.
python3 -m pip install clearml-agent==0.17.2rc3

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					DeliciousBluewhale87
				
					0
					 × 1

Is the agent itself registered on the clearml-server (a.k.a can you see it in the UI?)

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Show more results

Write your answer

113K Views

34 Answers

4 years ago

one year ago

Tags

Similar posts

Hpo With Optuna Via Webapp (Pro Version) Is Not Working As Expected. I Generated A Dummy Task That Logs A Random Value Into Some Metric And Closes, The Code:

Hi Guys! Im Running Into An Issue... Im Trying To Do My Master Thesis And Clearml Is A Framework Im Investing A Lot In. And Rn Im Trying To Set Up The Serving Module Into The Server, But If I Run It On The Server I Get Connection Issues Using External Ip+

Hi Guys, Im Running Into An Issue When Creating New Clearml Dataset Version. I Want To Create New Version Of Dataset From A Local Folder, But I Don'T Want All Of The Files In The Folder To Be Included, So I Cannot Use

Within A Clearml Task, Is It Possible To Checkout Another Task Code ? Context: I Have A Bit Of A Special Use-Case: We Train Our Model Using Clearml. Our Code Prep The Data: Download Images (We Are Not Using Clearml Dataset) And Bundle Labels Into A Dataf

Hey All, I'Ve Been Doing Some Hyperparameter Tuning Today Using A Clearml Managed Dataset. However, I Kept Running Into The Following Error:

Hi, I'Ve Run Into A Problem And Would Appreciate Some Help. I Installed Clearml Locally. When I Run A New Task On A Remote Server And In The Python Training Code I Set It To Only Train On One Gpu. Everything Works Fine And I See All The Scalars Automatica

Hello Everyone! I Am Relatively New To Clearml And To The *-Ops Concepts At All, As I Am But A Regular Python Dev. I Am Currently Trying To Implement Mlops Into Our Existing Local Infrastructure, So That We Would Be Able To Utilize Automated Data Preproc

Hi All Im Trying To Save My Model Checkpoints During Runtime But Am Running Into A Confusing Snag. I'M Using The Huggingface Architecture For A Transformer. Using Their Training Module To Control Training. In The Training Args, I Have The

Hi Team, I Am Trying To Start The Clearml With Opensearch As Backend Instead Elastic Search , When I Started The Service I Am Seeing Partial Web Ui As Below Kindly Help Me In Which Module I Should Look Into It To Get Complete Functionality. No Errors I A

As Soon As I Refactor My Project Into Multiple Folders, Where On Top-Level I Put My Pipeline File, And Keep My Tasks In A Subfolder, The Clearml Agent Seems To Have Problems: