ClearML FAQ | I Just Deployed Clearml Into K8 Cluster Using Clearml Helm Package. When I Ran A Job, It Gave This Error In The Clearml Web Server (Attached Below). I Sshed Into The Pod Running The Clearml-Agent. Upon Typing Clearml-Agent Init, I Realised The Clearml.Con

Answered

I Just Deployed Clearml Into K8 Cluster Using Clearml Helm Package. When I Ran A Job, It Gave This Error In The Clearml Web Server (Attached Below). I Sshed Into The Pod Running The Clearml-Agent. Upon Typing Clearml-Agent Init, I Realised The Clearml.Con

I just deployed clearml into k8 cluster using clearml helm package. When i ran a job, it gave this error in the clearml web server (Attached below). I sshed into the pod running the clearml-agent. Upon typing clearml-agent init, i realised the clearml.conf is empty and also I am not able to configure it (it is write protected). I am only able to configure the clearml.conf in the agentservices node.
So i am unable to run a clearml-agent within the k8 system, however, things are working if the clearml-agent is outisde the k8 system (my laptop). Do let me know if there is some way to debug this ?

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					DeliciousBluewhale87
				
					0
					 × 1

Votes Newest

Answers 34

It might be that the worker was killed before unregistered, you will see it there but the last update will be stuck (after 10min it will be automatically removed)

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Ohhh yes, that the problem

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					DeliciousBluewhale87
				
					0
					 × 1

Hi DeliciousBluewhale87
clearml-agent 0.17.2 was just release with the fix, let me know if it works

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Show more results

Write your answer

123K Views

34 Answers

4 years ago

one year ago

Tags

Similar posts

Hello, I Am Running Into An Issue With Clearml Pipelines I Have A Data_Prepping Step Which Contains A Lightningdatamodule. In It, I Load The Data And Prep It. My Function Then Returns An Initialized Datamodule Which I Give To The Training Function. I Hav

Hi Team, How Would You, Or What Is The Clearml Recommended Practice For Organising Sub-Steps Inside Each Pipeline Step? For Example I Have Several Sources Of Data To Retrieve, Each Of Which Is Potentially A Sub Step As I Wanted To Group Them Into One Big

Hpo With Optuna Via Webapp (Pro Version) Is Not Working As Expected. I Generated A Dummy Task That Logs A Random Value Into Some Metric And Closes, The Code:

Hi Guys! Im Running Into An Issue... Im Trying To Do My Master Thesis And Clearml Is A Framework Im Investing A Lot In. And Rn Im Trying To Set Up The Serving Module Into The Server, But If I Run It On The Server I Get Connection Issues Using External Ip+

Hi Guys, Im Running Into An Issue When Creating New Clearml Dataset Version. I Want To Create New Version Of Dataset From A Local Folder, But I Don'T Want All Of The Files In The Folder To Be Included, So I Cannot Use

Within A Clearml Task, Is It Possible To Checkout Another Task Code ? Context: I Have A Bit Of A Special Use-Case: We Train Our Model Using Clearml. Our Code Prep The Data: Download Images (We Are Not Using Clearml Dataset) And Bundle Labels Into A Dataf

Hey All, I'Ve Been Doing Some Hyperparameter Tuning Today Using A Clearml Managed Dataset. However, I Kept Running Into The Following Error:

Hi, I'Ve Run Into A Problem And Would Appreciate Some Help. I Installed Clearml Locally. When I Run A New Task On A Remote Server And In The Python Training Code I Set It To Only Train On One Gpu. Everything Works Fine And I See All The Scalars Automatica

Hello Everyone! I Am Relatively New To Clearml And To The *-Ops Concepts At All, As I Am But A Regular Python Dev. I Am Currently Trying To Implement Mlops Into Our Existing Local Infrastructure, So That We Would Be Able To Utilize Automated Data Preproc

Hi All Im Trying To Save My Model Checkpoints During Runtime But Am Running Into A Confusing Snag. I'M Using The Huggingface Architecture For A Transformer. Using Their Training Module To Control Training. In The Training Args, I Have The