Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Am Using Clearml Pro And Pretty Regularly I Will Restart An Experiment And Nothing Will Get Logged To Clearml. It Shows The Experiment Running (For Days) And It'S Running Fine On The Pc But No Scalers Or Debug Samples Are Shown. How Do We Troubleshoot T

I am using ClearML Pro and pretty regularly I will restart an experiment and nothing will get logged to ClearML. It shows the experiment running (for days) and it's running fine on the PC but no scalers or debug samples are shown.
How do we troubleshoot this?

  
  
Posted 3 months ago
Votes Newest

Answers 69


That makes sense... If you turn auto_connect_streams to false this mean that auto reporting will be disabled as per the documentation.. If you turn it to True then logging should resume.

  
  
Posted 3 months ago

When I try to abort an experiment. I get this in the log

clearml.Task - WARNING - ### TASK STOPPED - USER ABORTED - STATUS CHANGED ###

but it does not stop anything it just continues to run

  
  
Posted 3 months ago

Does any exit code appear? What is the status message and status reason in the 'INFO' section?

  
  
Posted 3 months ago

Do you also see the same in the terminal itself on the machine?

  
  
Posted 3 months ago

Yes I see it in the terminal on the machine

  
  
Posted 3 months ago

I do have uncommitted code changes. I can try to check at some point if it would not have the problem without them. It seems like it could be repeated just by making a git repo with that script and adding a very large file. If I can repeat it is it best to open an issue in GitHub?

  
  
Posted 3 months ago

Thank you @<1719524641879363584:profile|ThankfulClams64> for opening the GI, hopefully we will be able to reproduce it and fox ot quickly

  
  
Posted 3 months ago

I created an issue: None

  
  
Posted 3 months ago

I'm not sure how to even troubleshoot this.

  
  
Posted 3 months ago

It was working for me. Anyway I modified the callback. Attached is the script that has the issue for me whenever I add random_image_logger to the callbacks It only logs some of the scalars for 1 epoch. It then is stuck and never recovers. When I remove random_image_logger the scalars are correctly logged. Again this only on 1 computer, other computers we have logging work perfectly fine

  
  
Posted 3 months ago

sometimes I get no scalars, but the console logging always seems to be working

  
  
Posted 3 months ago

@<1719524641879363584:profile|ThankfulClams64> , are logs showing up without issue on the 'problematic' machine?

  
  
Posted 3 months ago

Yes it is logging to the console. The script does hang whenever it completes all the epochs when it is having the issue.

  
  
Posted 3 months ago

When the script is hung at the end the experiment says failed in ClearML

  
  
Posted 3 months ago

There is clearly some connection to the ClearML server as it remains "running" the entire training session but there are no metrics or debug samples. And I see nothing in the logs to indicate there is an issue

  
  
Posted 3 months ago

This was on the same machine I am having issues with it logs scalars correctly using the example code, but when I add in that callback which just logs a random image to tensorboard I don't get any scalars logged

  
  
Posted 3 months ago

So I am only seeing values for the first epoch. It seems like it does not track all of them so maybe something is happening when it tries to log scalars.
I have seen it only log iterations but setting task.set_initial_iteration(0) seemed to fix that so it now seems to be logging the correct epoch
Tensorboard is correct and works. I have never seen an issue in the tensorboard logs

  
  
Posted 3 months ago

It is not always reproducible it seems like something that we do not understand happens then the machine consistently has this issue. We believe it has something to do with stopping and starting experiments

  
  
Posted 3 months ago

I'm not sure if it still reports logs. But it will continue running on the machine

  
  
Posted 3 months ago

Console logs

  
  
Posted 3 months ago

The machine currently having the issue is on tensorboard==2.16.2

  
  
Posted 3 months ago

Not sure why that is related to saving images

  
  
Posted 3 months ago

STATUS MESSAGE: N/A
STATUS REASON: Signal None

  
  
Posted 3 months ago

Is there someway to kill all connections of a machine to the ClearML server this does seem to be related to restarting a task / running a new task quickly after a task fails or is aborted

  
  
Posted 3 months ago

My bad, if you set auto_connect_streams to false, you basically disable the console logging... Please see the documentation:

auto_connect_streams (Union[bool, Mapping[str, bool]]) – Control the automatic logging of stdout and stderr.
  
  
Posted 3 months ago

Yes it shows on the UI and has the first epoch for some of the metrics but that's it. It has run like 50 epochs, it says it is still running but there are no updates to the scalars or debug samples

  
  
Posted 3 months ago

Can you share any of the logs?

  
  
Posted 3 months ago

I'll update my clearml version. Unfortunately I do not have a small code snippet and it is not always repeatable. Is there some additional logging that can be turned on?

  
  
Posted 3 months ago

Yes tensorboard. It is still logging the tensorboard scalers and images. It just doesn't log the console output

  
  
Posted 3 months ago

Console output and also what you get on the ClearML task page under the console section

  
  
Posted 3 months ago
7K Views
69 Answers
3 months ago
2 months ago
Tags