![Profile picture](https://clearml-web-assets.s3.amazonaws.com/scoold/avatars/ThankfulClams64.png)
Reputation
Badges 1
49 × Eureka!Running clearml_example.py in None reproduces the issue
I do have uncommitted code changes. I can try to check at some point if it would not have the problem without them. It seems like it could be repeated just by making a git repo with that script and adding a very large file. If I can repeat it is it best to open an issue in GitHub?
When I try to abort an experiment. I get this in the log
clearml.Task - WARNING - ### TASK STOPPED - USER ABORTED - STATUS CHANGED ###
but it does not stop anything it just continues to run
STATUS MESSAGE: N/A
STATUS REASON: Signal None
I didn't do a very scientific comparison but the # of API calls did decrease substantially by turning off auto_connect_streams
It is probably about 100k API calls per day with 1 experiment running where before it was maybe 300k API calls per day. Still seems like a lot when I only run 20-30 epochs in a day
We are running the same code on multiple machines and it just randomly happens. Currently we are having the issue on 1 out of 4
sometimes I get no scalars, but the console logging always seems to be working
Is this just the console output while training?
So I was able to repeat the same behavior on a machine running this example None
by adding the following callback
class TensorBoardImage(TensorBoard):
@staticmethod
def make_image(tensor):
from PIL import Image
import io
tensor = np.stack((tensor, tensor, tensor), axis=2)
height, width, channels = tensor.shape
image = Image.from...
I just created a new virtual environment and the problem persists. There are only two dependencies clearml and tensorflow. @<1523701070390366208:profile|CostlyOstrich36> what logs are you referring to?
Another thing I notice is that aborting the experiment does not work when this is happening. It just continues to run
I will try with clearml==1.16.3rc2 and see if it still has the issue
Not sure if this is helpful but this is what I get when I cntrl-c out of the hung script
^C^CException ignored in atexit callback: <bound method Reporter._handle_program_exit of <clearml.backend_interface.metrics.reporter.Reporter object at 0x70fd8b7ff1c0>>
Event reporting sub-process lost, switching to thread based reporting
Traceback (most recent call last):
File "/home/richard/.virtualenvs/temp_clearml/lib/python3.10/site-packages/clearml/backend_interface/metrics/reporter.py", lin...
The machine currently having the issue is on tensorboard==2.16.2
Yes I see it in the terminal on the machine
No it completes and exists the script
I'm not sure if it still reports logs. But it will continue running on the machine
I am on 1.16.2
task = Task.init(project_name=model_config['ClearML']['project_name'],
task_name=model_config['ClearML']['task_name'],
continue_last_task=False,
auto_connect_streams=True)
It is not always reproducible it seems like something that we do not understand happens then the machine consistently has this issue. We believe it has something to do with stopping and starting experiments
Yes it is logging to the console. The script does hang whenever it completes all the epochs when it is having the issue.
I found that setting store_uncommitted_code_diff: false
instead of true seems to fix the issue
So I am only seeing values for the first epoch. It seems like it does not track all of them so maybe something is happening when it tries to log scalars.
I have seen it only log iterations but setting task.set_initial_iteration(0)
seemed to fix that so it now seems to be logging the correct epoch
Tensorboard is correct and works. I have never seen an issue in the tensorboard logs
Not sure why that is related to saving images
It seems similar to this None is it possible saving too many model weights causes metric logging thread to die?
It was working for me. Anyway I modified the callback. Attached is the script that has the issue for me whenever I add random_image_logger
to the callbacks It only logs some of the scalars for 1 epoch. It then is stuck and never recovers. When I remove random_image_logger
the scalars are correctly logged. Again this only on 1 computer, other computers we have logging work perfectly fine
I'm not sure how to even troubleshoot this.
Then we also connect two dictionaries for configs
task.connect(model_config)
task.connect(DataAugConfig)
When the script is hung at the end the experiment says failed in ClearML
I am still having this issue. An update is that the "abort" does not work. Even though the state is correctly tracked in ClearML when I try to abort the experiment through the UI it says it does it but the experiment remains running on the computer.