I’m also curious if it’s available to bind the same GPU to multiple queues.

Thanks! also logs too?

thank you CostlyOstrich36

pls also refer to None :)

Oh, it didn’t generate conf file properly. I will try again

hello CostlyOstrich36 unfortunately, i also did it to api server just in case. but didn’t work

I want to get task id, properties right after submitting clearml-session task

@<1523701205467926528:profile|AgitatedDove14> Good! I will try it

I tried using K8S_GLUE_POD_AGENT_INSTALL_ARGS=1.5.3rc2 instead of CLEARML_AGENT_UPDATE_VERSION=1.5.3rc2 , but it’s same. doesn’t read gpu usage.. 🥲

Hope clearml-session will be more developed as clearml-agent. cause it is so useful! 👍
My issue: None

Wow i appreciate that 😆

Hi @<1523701205467926528:profile|AgitatedDove14>
The server is already self hosted. I realized i can’t create a report using clearml sdk. so i think i need to find other ways

then, is there any way to get embed code from scalars?

i understand the reason that clearml-session supports only cli is because of SSH. right? i thought it was easy to develop sdk. instead, i can use your recommendation

for more info, I set CLEARML_AGENT_UPDATE_VERSION=1.5.3rc2 ` in agentk8sglue.basePodTemplate.env

because clearml-agnet is not installed in my gke cluster

It seems that there is no way to add environments, so i customized charts and using it on my own.

@<1523701087100473344:profile|SuccessfulKoala55> I realized that this is not an issue with the cloud or on-premise environment. it’s working well on gke but not working on eks. here is the log when i run “clearml-agent daemon --queue ~” command on eks

root@shelley-gpu-pod:/# clearml-agent daemon --queue shelley3
/usr/local/lib/python3.8/dist-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.0.1) or chardet (None)/charset_normalizer (3.1.0) doesn’t match a supported ve...

root@shelley-gpu-pod:/# clearml-agent daemon --queue shelley2 --foreground
/usr/local/lib/python3.8/dist-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.0.2) or chardet (None)/charset_normalizer (3.1.0) doesn’t match a supported version!
warnings.warn(
Using environment access key CLEARML_API_ACCESS_KEY=“”
Using environment secret key CLEARML_API_SECRET_KEY=********
Current configuration (clearml_agent v1.5.2, location: None):

agent.worker_id ...

I tried the suggestion you mentioned, but it’s the same. And it doesn’t seem to be an AMI issue. The same problem is occurring even in an on-premise environment.

here is the agent, task log file~!

any updates?..

Hi again 😊 @<1523701087100473344:profile|SuccessfulKoala55> sure!

@<1523701087100473344:profile|SuccessfulKoala55> what is task log? you mean the pod log provisioned by clearml-agent? do you want me to show them?

@<1523701087100473344:profile|SuccessfulKoala55> yes. It only occurs when running on the cloud. It’s fine when running on-premises.

pod log is too long. would it be ok if i upload pod log file here??

I set CLEARML_AGENT_UPDATE_VERSION=1.5.3rc2 ` in agentk8sglue.basePodTemplate.env as i mentioned

Oh, It’s not the issue with eks.. We had the same issue on an on-premise cluster too(clearml-agent is installed). Could it be because of clearml-agent installed?

It also shows on project detail page.