Wow i appreciate that 😆

it is working on on-premise machine(i can see gpu usage on WORKERS & QUEUES Dashboard). but it is not working on cloud pod

nope. just running “clearml-agent daemon --queue shelley”

heres is the log when executing with --foreground. but is there any difference?

@<1523701087100473344:profile|SuccessfulKoala55> what is task log? you mean the pod log provisioned by clearml-agent? do you want me to show them?

Thanks! also logs too?

then, is there any way to get embed code from scalars?

It seems that there is no way to add environments, so i customized charts and using it on my own.

Oh, It’s not the issue with eks.. We had the same issue on an on-premise cluster too(clearml-agent is installed). Could it be because of clearml-agent installed?

I set CLEARML_AGENT_UPDATE_VERSION=1.5.3rc2 ` in agentk8sglue.basePodTemplate.env as i mentioned

thank you CostlyOstrich36

Hi @<1523701205467926528:profile|AgitatedDove14>
The server is already self hosted. I realized i can’t create a report using clearml sdk. so i think i need to find other ways

Hi again 😊 @<1523701087100473344:profile|SuccessfulKoala55> sure!

i fount the solution!! i added configuration to helm’s values.yaml below.

additionalConfigs:
# services.conf: |
# tasks {
# non_responsive_tasks_watchdog {
# # In-progress tasks that haven’t been updated for at least ‘value’ seconds will be stopped by the watchdog
# threshold_sec: 21000
# # Watchdog will sleep for this number of seconds after each cycle
# watch_interval_sec: 900
# }
# }
apiserver.co...

hello CostlyOstrich36 unfortunately, i also did it to api server just in case. but didn’t work

I’m also curious if it’s available to bind the same GPU to multiple queues.

can i hide some of them without fixing and rebuilding docker image?

i understand the reason that clearml-session supports only cli is because of SSH. right? i thought it was easy to develop sdk. instead, i can use your recommendation

I want to get task id, properties right after submitting clearml-session task

sure! this one?

@<1523701087100473344:profile|SuccessfulKoala55> Okay..but how can i specify agent’s verison in helm chart?

pls also refer to None :)

root@shelley-gpu-pod:/# clearml-agent daemon --queue shelley2 --foreground
/usr/local/lib/python3.8/dist-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.0.2) or chardet (None)/charset_normalizer (3.1.0) doesn’t match a supported version!
warnings.warn(
Using environment access key CLEARML_API_ACCESS_KEY=“”
Using environment secret key CLEARML_API_SECRET_KEY=********
Current configuration (clearml_agent v1.5.2, location: None):

agent.worker_id ...

it’s been working well until i removed virtualenv and recreated, then i reinstall only clearml and clearml-session

I tried the suggestion you mentioned, but it’s the same. And it doesn’t seem to be an AMI issue. The same problem is occurring even in an on-premise environment.

i am having same issue: None

clearml==1.9.2
clearml-session==0.4.0

@<1523701087100473344:profile|SuccessfulKoala55> yes. It only occurs when running on the cloud. It’s fine when running on-premises.