Console output and also what you get on the ClearML task page under the console section
ThankfulClams64 you could try using the compare function in the UI to compare the experiments on the machine the scalars are not reported properly and the experiments on a machine that runs the experiments properly. I suggest then replicating the environment exactly on the problematic machine. None
task.connect(model_config)
task.connect(DataAugConfig)
If these are separate dictionaries , you should probably use two sections:
task.connect(model_config, name="model config")
task.connect(DataAugConfig, name="data aug")
It is still getting stuck.
I notice that one of the scalars that gets logged early is logging the epoch while the remaining scalars seem to be iterations because the iteration value is 1355 instead of 26
wait so you are seeing Some scalars ?
while the remaining scalars seem to be iterations because the iteration value is 1355 instead of 26
what are you seeing in your TB?
Just to make sure, did the logging to the clearml server work previously and stoped working at some point?
The console logging still works. Aborting the task was in the log but did not work and the process continued until I killed it.
Correct, so I get something like this
ClearML Task: created new task id=6ec57dcb007545aebc4ec51eb5b34c67
======> WARNING! Git diff too large to store (2536kb), skipping uncommitted changes <======
ClearML results page:
but that is all
My bad, if you set auto_connect_streams to false, you basically disable the console logging... Please see the documentation:
auto_connect_streams (Union[bool, Mapping[str, bool]]) – Control the automatic logging of stdout and stderr.
Hi ThankfulClams64 ,the logging is done by a separate process, I'm pretty sure it's not terminating all of the sudden. Did you manage to get a full log of such an experiment to share?
ThankfulClams64 , are logs showing up without issue on the 'problematic' machine?
Thank you ThankfulClams64 for opening the GI, hopefully we will be able to reproduce it and fox ot quickly
We are running the same code on multiple machines and it just randomly happens. Currently we are having the issue on 1 out of 4
If you remove any reference of ClearML from the code on that machine, does it still hang?
Can you try with auto_connect_streams=True ? Also, what version of clearml sdk are you using?
Hi ThankfulClams64 , stopping all processes should do that, there is no programmatic way of doing that specifically. Did you try calling task.close() for all tasks you're using?
The machine currently having the issue is on tensorboard==2.16.2
I am using ClearML Pro and pretty regularly I will restart an experiment and nothing will get logged to ClearML.
I use ClearML with pytorch 1.7.1, pytorch-lightning 1.2.2 and Tensorboard auto
All ClearML has the latest stable updates. (clearml 1.7.4, clearml-agent 1.7.2)
Is this still happening with the latest clearml ( clearml==1.16.3rc2 ) ?
What is the TB version?
I remember a fix regrading lightining support
Also just making sure, are you using the default lightning TB logger ?
How are you initializing the Task.init (i.e. could you copy here the code?)
It is not always reproducible it seems like something that we do not understand happens then the machine consistently has this issue. We believe it has something to do with stopping and starting experiments
ThankfulClams64 , can you provide a small code snippet that reproduces this behaviour? Can you also test with the latest version of clearml ?
Do you also see the same in the terminal itself on the machine?
I am using 1.15.0. Yes I can try with auto_connect_streams set to True I believe I will still have the issue
Hi ThankfulClams64 , does the experiment itself show on the ClearML UI?
Then we also connect two dictionaries for configs
task.connect(model_config)
task.connect(DataAugConfig)
I found that setting store_uncommitted_code_diff: false instead of true seems to fix the issue
It was working for me. Anyway I modified the callback. Attached is the script that has the issue for me whenever I add random_image_logger to the callbacks It only logs some of the scalars for 1 epoch. It then is stuck and never recovers. When I remove random_image_logger the scalars are correctly logged. Again this only on 1 computer, other computers we have logging work perfectly fine
Yea I am fine not having the console logging. My issues is the scalers and debug images occasionally don't record to ClearML
Hi we are currently having the issue. There is nothing in the console regarding ClearML besides
ClearML Task: created new task id=0174d5b9d7164f47bd10484fd268e3ff
======> WARNING! Git diff too large to store (3611kb), skipping uncommitted changes <======
ClearML results page:
The console logs continue to come in put no scalers or debug images show up.
That makes sense... If you turn auto_connect_streams to false this mean that auto reporting will be disabled as per the documentation.. If you turn it to True then logging should resume.
ThankfulClams64 , if you set auto_connect_streams to false nothing will be reported from your frameworks. With what frameworks are you working, tensorboard?
It seems similar to this None is it possible saving too many model weights causes metric logging thread to die?