So it is the automagic that is not working.
Can you print the following before calling Both Task.debug_simulate_remote_task
and Task.init
, Notice you have to call Task.initprint(os.environ)
maybe another warning here? https://clear.ml/docs/latest/docs/references/sdk/task/
the situation is:
I can report a scalar by calling clearml_task.logger.report_scalar
and it becomes immediately visible in the web ui
report_scalar works, report_text does not, this is very weird
I can't make anything appear in the console part of the ui
but seriously, I am very thankful you were willing to spend so much time helping me, I am super impressed by your response time and helpfulness!
no, I set the env variable CLEARML_TASK_ID myself
Do not, this is the issue 🙂
this is used internally and messing up the internal state, basically this is one of the signals for the SDK to know there is an agent taking care of things (for example logging the entire console output)
Use any other variable, for example MY_CLEARML_TASK_ID
maybe not at the top but in the Task.init
description
I could have been more inventive as well 😄
I guess we should have obfuscated the name better 😄
Just to let you know, it now works (obviously) in the k8s setting as well.
report_text does not, this is very weird
Okay this seems to be the issue.
Just making sure the Task status is "running" and task.get_logger().report_text("something")
does not report a thing ?
Do you see it on your screen?
Can you test without the "Task.debug_simulate_remote_task / init" ?
I don't know whether you have access to the backend,
Creepy , no I do not 🙂
I can't make anything appear in the console part of the ui
clearml_task.logger.report_text("some text")
should work
but perhaps it is worth adding to the docs page a hint to avoid using the CLEARML_TASK_ID env variable, perhaps I am not the only one to ever try it
Good idea, any thoughts on where ? I cannot find a trivial place to put these things
console output:ClearML results page:
01b77a220869442d80af42efce82c617 some text 2022-03-21 22:47:16,660 - clearml.Task - INFO - Waiting to finish uploads 2022-03-21 22:47:28,217 - clearml.Task - INFO - Finished uploading
no, I set the env variable CLEARML_TASK_ID myself
now it stopped working locally as well
At least this is consistent 🙂
How so ? Is the "main" Task still running ?
my code snippet
` from clearml import Task
import os
clearml_task_id = os.environ['CLEARML_TASK_ID']
Task.debug_simulate_remote_task(clearml_task_id)
clearml_task = Task.init(auto_connect_arg_parser=False, auto_resource_monitoring=False)
print(clearml_task.id)
clearml_task.logger.report_scalar(series='s', value='123', iteration=2, title='title')
clearml_task.logger.report_text("some text") `
Wait, how did you end up withclearml_task_id = os.environ['CLEARML_TASK_ID']
printing "01b77a220869442d80af42efce82c617" ?
This means you are running by an agent?!
yes, I don't know whether you have access to the backend, but just in case my experiment is this one:
ClearML results page: https://app.clear.ml/projects/45557a1ee1464631a9a18b0dcda4f682/experiments/01b77a220869442d80af42efce82c617/output/log
but perhaps it is worth adding to the docs page a hint to avoid using the CLEARML_TASK_ID env variable, perhaps I am not the only one to ever try it
I can hardcode it into program if you want