I created my own docker image with a newer python and the error disappeared

yes, those

I subscribe to the problem of having large metrics without a tool for proper inspection what is it coming from.

I am not an expert on this, just started using torchmetrics.

they are universal, I thought there is some interface to them in clearml, but probably not

ok

thanks!

I think there was some problem how shutil.copytree works in python3.6 with broken links

ok, I solved the problem,
agent.force_git_ssh_protocol = truedid the trick

I did not configure user/pass for git

my code snippet
` from clearml import Task
import os

clearml_task_id = os.environ['CLEARML_TASK_ID']
Task.debug_simulate_remote_task(clearml_task_id)
clearml_task = Task.init(auto_connect_arg_parser=False, auto_resource_monitoring=False)
print(clearml_task.id)
clearml_task.logger.report_scalar(series='s', value='123', iteration=2, title='title')
clearml_task.logger.report_text("some text") `

it is a configuration object (line of my code:
config_path = task.connect_configuration(config_path)

got it!

but it is a guess

ok, but do you know why did it try to reuse in the first place?

ok, understood, it was probably my fault, I was messing up with the services container and probably made the pipeline task interrupted, so the subtasks themselves have finished, but the pipeline task was not alive when it happened

(not that it is complicated)

I did not know about it, thanks!

Sure, I can polish my code of corse, but wanted to first check if there is to be any useful.

I did something similar to what you suggests and it worked, the key insight was that connect and connect_configuration work differently in terms of overrides, thanks!

just put ssh config with the proper key marked

traceback:
` Traceback (most recent call last):
File "/home/marek/nomagic/monomagic/ml/tiresias/calibrate_and_test.py", line 57, in <module>
Task.add_requirements('requirements.txt')
File "/home/marek/.virtualenvs/tiresias-3.9/lib/python3.9/site-packages/clearml/backend_interface/task/task.py", line 1976, in add_requirements
for req in pkg_resources.parse_requirements(requirements_txt):
File "/home/marek/.virtualenvs/tiresias-3.9/lib/python3.9/site-packages/pkg_resources/_init...

this is part of repository

I circumvented the problem by putting timestamp in task name, but I don't think this is necessary.

@<1523701435869433856:profile|SmugDolphin23> will send later today

@<1523701435869433856:profile|SmugDolphin23> it did not help, shall I create smallest example when it does not work and paste it here?

yes, thanks!

@<1523701087100473344:profile|SuccessfulKoala55> FYI

👍

I am only getting one user for some reason, even though 4 are in the system