I just deployed clearml into k8 cluster using clearml helm package. When i ran a job, it gave this error in the clearml web server (Attached below). I sshed into the pod running the clearml-agent. Upon typing clearml-agent init, i realised the clearml.conf is empty and also I am not able to configure it (it is write protected). I am only able to configure the clearml.conf in the agentservices node.
So i am unable to run a clearml-agent within the k8 system, however, things are working if the clearml-agent is outisde the k8 system (my laptop). Do let me know if there is some way to debug this ?
This is from my k8 cluster. Using the clearml helm package, I was able to set this up.
It might be that the worker was killed before unregistered, you will see it there but the last update will be stuck (after 10min it will be automatically removed)
Yeah, I restarted the deployment and sshed into the host machine also.. (Img below)
