I am seeing such warnings clearml.model - WARNING - 9 model found when searching

@<1523701087100473344:profile|SuccessfulKoala55> any ideas what can be the cause?

I could have been more inventive as well 😄

yes, I am calling Task.init

ok, understood, it was probably my fault, I was messing up with the services container and probably made the pipeline task interrupted, so the subtasks themselves have finished, but the pipeline task was not alive when it happened

I did something similar to what you suggests and it worked, the key insight was that connect and connect_configuration work differently in terms of overrides, thanks!

it is a configuration object (line of my code:
config_path = task.connect_configuration(config_path)

I did not configure user/pass for git

there is a broken symlink in the original repository

thanks!

I created my own docker image with a newer python and the error disappeared

console output:
ClearML results page: 01b77a220869442d80af42efce82c617 some text 2022-03-21 22:47:16,660 - clearml.Task - INFO - Waiting to finish uploads 2022-03-21 22:47:28,217 - clearml.Task - INFO - Finished uploading

I can't make anything appear in the console part of the ui

@<1523701435869433856:profile|SmugDolphin23> it did not help, shall I create smallest example when it does not work and paste it here?

the situation is:
I can report a scalar by calling clearml_task.logger.report_scalar and it becomes immediately visible in the web ui

I did not know about it, thanks!

We have a training template that is a k8s job definition (yaml) that creates env variables inside the docker images that is used for tranining, and those env variables are credentials for ClearML. Since they are taken from k8s secrets, they are the same for every user.

I can create secrets for every new user and set env variables accordingly, but perhaps you see a better way out?

@<1523701435869433856:profile|SmugDolphin23> will send later today

and in the future I do want to have an Agent on the k8s cluster, but then this should not be a problem I guess as the user is set during Task.init , right?

SuccessfulKoala55 that worked, thanks a lot!

Is there a place where I can find details about this approach?

@<1523701435869433856:profile|SmugDolphin23> I have checked that when setting auto_connect_frameworks=False it works, but disabling just joblib is not enough.

From the documentation https://github.com/allegroai/clearml-agent :
` Two K8s integration flavours

Spin ClearML-Agent as a long-lasting service pod
use clearml-agent docker image
map docker socket into the pod (soon replaced by podman)
allow the clearml-agent to manage sibling dockers
benefits: full use of the ClearML scheduling, no need to worry about wrong container images / lost pods etc.
downside: Sibling containers `

I circumvented the problem by putting timestamp in task name, but I don't think this is necessary.

yes, on-prem

no, I set the env variable CLEARML_TASK_ID myself

I can hardcode it into program if you want

this is part of repository

my code snippet
` from clearml import Task
import os

clearml_task_id = os.environ['CLEARML_TASK_ID']
Task.debug_simulate_remote_task(clearml_task_id)
clearml_task = Task.init(auto_connect_arg_parser=False, auto_resource_monitoring=False)
print(clearml_task.id)
clearml_task.logger.report_scalar(series='s', value='123', iteration=2, title='title')
clearml_task.logger.report_text("some text") `

now it stopped working locally as well