I just used CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL can that be put int he clearml.conf? I didn't see a reference to it in the documentaiton

I guess I don't understand I am referring to the clearml configuration file on the agent. The only way I have gotten it to consistently work is to just install the environment before hand and set that environment variable. Otherwise it seems clearml is not correctly saving the environment to be able to reproduce it. In my case the issues is installing tensorflow instead of tensorflow[and-cuda] which is what was installed

Yes it is logging to the console. The script does hang whenever it completes all the epochs when it is having the issue.

I just created a new virtual environment and the problem persists. There are only two dependencies clearml and tensorflow. @<1523701070390366208:profile|CostlyOstrich36> what logs are you referring to?

There is clearly some connection to the ClearML server as it remains "running" the entire training session but there are no metrics or debug samples. And I see nothing in the logs to indicate there is an issue

Yes I see it in the terminal on the machine

Yes tensorboard. It is still logging the tensorboard scalers and images. It just doesn't log the console output

Another thing I notice is that aborting the experiment does not work when this is happening. It just continues to run

So I was able to repeat the same behavior on a machine running this example None

by adding the following callback

class TensorBoardImage(TensorBoard):
    @staticmethod
    def make_image(tensor):
        from PIL import Image
        import io
        tensor = np.stack((tensor, tensor, tensor), axis=2)
        height, width, channels = tensor.shape
        image = Image.from...

Not sure why that is related to saving images

I'm not sure how to even troubleshoot this.

When I try to abort an experiment. I get this in the log

clearml.Task - WARNING - ### TASK STOPPED - USER ABORTED - STATUS CHANGED ###

but it does not stop anything it just continues to run

Yea, from all the YouTube videos it is just there with no mention of how to get it. But I don't have it

The same training works sometimes. But I'm not sure how to troubleshoot when it stops logging the metrics

Thank you! I think that is all I need to do

Okay I will do another run to capture the console output. We currently set auto_connect_streams to False to reduce the number of API calls. So there isn't really anything in the ClearML task page console section

When the script is hung at the end the experiment says failed in ClearML

Running clearml_example.py in None reproduces the issue

It seems similar to this None is it possible saving too many model weights causes metric logging thread to die?

We are running the same code on multiple machines and it just randomly happens. Currently we are having the issue on 1 out of 4

Yea I am fine not having the console logging. My issues is the scalers and debug images occasionally don't record to ClearML

Do the metrics not get added from the training? I did not add any metadata data but I assumed you would be able to select metrics from the training that generated the model

This was on the same machine I am having issues with it logs scalars correctly using the example code, but when I add in that callback which just logs a random image to tensorboard I don't get any scalars logged

Is this just the console output while training?

So I am only seeing values for the first epoch. It seems like it does not track all of them so maybe something is happening when it tries to log scalars.
I have seen it only log iterations but setting task.set_initial_iteration(0) seemed to fix that so it now seems to be logging the correct epoch
Tensorboard is correct and works. I have never seen an issue in the tensorboard logs

How do you get answers to these types of questions? As far as I can tell model registries is broken, and there is no support through the actual application

Okay thanks!

It was working for me. Anyway I modified the callback. Attached is the script that has the issue for me whenever I add random_image_logger to the callbacks It only logs some of the scalars for 1 epoch. It then is stuck and never recovers. When I remove random_image_logger the scalars are correctly logged. Again this only on 1 computer, other computers we have logging work perfectly fine

I have file_history_size: 1000 I still get images for following epochs. But sometimes it seems like in the UI it limits the view to 32 images.

They are tensorboard images that are autmomagically being logged to debug samples