now it stopped working locally as well
At least this is consistent 🙂
How so ? Is the "main" Task still running ?
yes, I don't know whether you have access to the backend, but just in case my experiment is this one:
ClearML results page: https://app.clear.ml/projects/45557a1ee1464631a9a18b0dcda4f682/experiments/01b77a220869442d80af42efce82c617/output/log
the situation is:
I can report a scalar by calling clearml_task.logger.report_scalar
and it becomes immediately visible in the web ui
I can't make anything appear in the console part of the ui
I don't know whether you have access to the backend,
Creepy , no I do not 🙂
I can't make anything appear in the console part of the ui
clearml_task.logger.report_text("some text")
should work
So it is the automagic that is not working.
Can you print the following before calling Both Task.debug_simulate_remote_task
and Task.init
, Notice you have to call Task.initprint(os.environ)
report_scalar works, report_text does not, this is very weird
report_text does not, this is very weird
Okay this seems to be the issue.
Just making sure the Task status is "running" and task.get_logger().report_text("something")
does not report a thing ?
Do you see it on your screen?
Can you test without the "Task.debug_simulate_remote_task / init" ?
my code snippet
` from clearml import Task
import os
clearml_task_id = os.environ['CLEARML_TASK_ID']
Task.debug_simulate_remote_task(clearml_task_id)
clearml_task = Task.init(auto_connect_arg_parser=False, auto_resource_monitoring=False)
print(clearml_task.id)
clearml_task.logger.report_scalar(series='s', value='123', iteration=2, title='title')
clearml_task.logger.report_text("some text") `
console output:ClearML results page:
01b77a220869442d80af42efce82c617 some text 2022-03-21 22:47:16,660 - clearml.Task - INFO - Waiting to finish uploads 2022-03-21 22:47:28,217 - clearml.Task - INFO - Finished uploading
Wait, how did you end up withclearml_task_id = os.environ['CLEARML_TASK_ID']
printing "01b77a220869442d80af42efce82c617" ?
This means you are running by an agent?!
no, I set the env variable CLEARML_TASK_ID myself
I can hardcode it into program if you want
no, I set the env variable CLEARML_TASK_ID myself
Do not, this is the issue 🙂
this is used internally and messing up the internal state, basically this is one of the signals for the SDK to know there is an agent taking care of things (for example logging the entire console output)
Use any other variable, for example MY_CLEARML_TASK_ID
I guess we should have obfuscated the name better 😄
I could have been more inventive as well 😄
but seriously, I am very thankful you were willing to spend so much time helping me, I am super impressed by your response time and helpfulness!
Just to let you know, it now works (obviously) in the k8s setting as well.
but perhaps it is worth adding to the docs page a hint to avoid using the CLEARML_TASK_ID env variable, perhaps I am not the only one to ever try it
but perhaps it is worth adding to the docs page a hint to avoid using the CLEARML_TASK_ID env variable, perhaps I am not the only one to ever try it
Good idea, any thoughts on where ? I cannot find a trivial place to put these things
maybe another warning here? https://clear.ml/docs/latest/docs/references/sdk/task/
maybe not at the top but in the Task.init
description