Reputation
Badges 1
49 × Eureka!I run clearml-agent manually in gpu available pod using command clearml-agent daemon --queue shelley
and this doesn’t show gpu usage same with when i run task remotely
and here is the log
agent.worker_id =
agent.worker_name = shelley-gpu-pod
agent.force_git_ssh_protocol = false
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version.0 = <20.2 ; python_version < ‘3.10’
agent.package_manager.pip_version.1 = <22.3 ; python_ver...
pod log is too long. would it be ok if i upload pod log file here??
@<1523701087100473344:profile|SuccessfulKoala55> yes. It only occurs when running on the cloud. It’s fine when running on-premises.
i am having same issue: None
Hi again 😊 @<1523701087100473344:profile|SuccessfulKoala55> sure!
I’m also curious if it’s available to bind the same GPU to multiple queues.
I tried using K8S_GLUE_POD_AGENT_INSTALL_ARGS=1.5.3rc2
instead of CLEARML_AGENT_UPDATE_VERSION=1.5.3rc2
, but it’s same. doesn’t read gpu usage.. 🥲
This is clearml-agent helm chart values.yaml file i used to install
heres is the log when executing with --foreground. but is there any difference?
Are there other people experiencing the same issue as me?
it’s been working well until i removed virtualenv and recreated, then i reinstall only clearml and clearml-session
@<1523701205467926528:profile|AgitatedDove14> Good! I will try it
I set CLEARML_AGENT_UPDATE_VERSION=1.5.3rc2
` in agentk8sglue.basePodTemplate.env as i mentioned
Thanks! also logs too?
Hi @<1523701205467926528:profile|AgitatedDove14>
The server is already self hosted. I realized i can’t create a report using clearml sdk. so i think i need to find other ways
i understand the reason that clearml-session supports only cli is because of SSH. right? i thought it was easy to develop sdk. instead, i can use your recommendation
@<1523701070390366208:profile|CostlyOstrich36> Hello. Oh, sorry for the lack of explanation.when i execute the command “clearml-session ~“, jupyter url format is ‘ None :{local_jupyter_port}/?token={jupyter_token}’ and vs code url format is just ‘ None :{local_vscode_port}’ like the pic i attached here. I wonder why vs code url doesn’t have token.