Reputation
Badges 1
75 × Eureka!SuccessfulKoala55 that worked, thanks a lot!
just put ssh config with the proper key marked
report_scalar works, report_text does not, this is very weird
ok, I solved the problem,agent.force_git_ssh_protocol = truedid the trick
console output:ClearML results page: 01b77a220869442d80af42efce82c617 some text 2022-03-21 22:47:16,660 - clearml.Task - INFO - Waiting to finish uploads 2022-03-21 22:47:28,217 - clearml.Task - INFO - Finished uploading
yes, I am calling Task.init
thanks! is this documented? (I am wondering whether I could have avoided bothering you with my question in the first place)
Sure, I can polish my code of corse, but wanted to first check if there is to be any useful.
it is typically sued with pytorch
it is a configuration object (line of my code:config_path = task.connect_configuration(config_path)
I did not configure user/pass for git
no, I set the env variable CLEARML_TASK_ID myself
they are universal, I thought there is some interface to them in clearml, but probably not
Is there a place where I can find details about this approach?
the situation is:
I can report a scalar by calling clearml_task.logger.report_scalar and it becomes immediately visible in the web ui
I did something similar to what you suggests and it worked, the key insight was that connect and connect_configuration work differently in terms of overrides, thanks!
ok, but do you know why did it try to reuse in the first place?
From the documentation https://github.com/allegroai/clearml-agent :
` Two K8s integration flavours
Spin ClearML-Agent as a long-lasting service pod
use clearml-agent docker image
map docker socket into the pod (soon replaced by podman)
allow the clearml-agent to manage sibling dockers
benefits: full use of the ClearML scheduling, no need to worry about wrong container images / lost pods etc.
downside: Sibling containers `
and in the future I do want to have an Agent on the k8s cluster, but then this should not be a problem I guess as the user is set during Task.init , right?
We have a training template that is a k8s job definition (yaml) that creates env variables inside the docker images that is used for tranining, and those env variables are credentials for ClearML. Since they are taken from k8s secrets, they are the same for every user.
I can create secrets for every new user and set env variables accordingly, but perhaps you see a better way out?
ok, I will do a simple workaround for this (use an additional parameter that I can update using parameter_override and then check if it exists and update the configuration in python myself)
but it is a guess