pod log is too long. would it be ok if i upload pod log file here??
@<1524922424720625664:profile|TartLeopard58> the agent running the task is v1.5.2 (as shown in the log), so the whole point is lost - we need to see the v1.5.3rc2 or v1.5.3rc3 running there... how did you set up the helm chart for the new agent?
This is clearml-agent helm chart values.yaml file i used to install
I set CLEARML_AGENT_UPDATE_VERSION=1.5.3rc2
` in agentk8sglue.basePodTemplate.env as i mentioned
Try using K8S_GLUE_POD_AGENT_INSTALL_ARGS=1.5.3rc2
I tried using K8S_GLUE_POD_AGENT_INSTALL_ARGS=1.5.3rc2
instead of CLEARML_AGENT_UPDATE_VERSION=1.5.3rc2
, but it’s same. doesn’t read gpu usage.. 🥲