BoredPigeon26 SuccessfulKoala55 CostlyOstrich36 Hi
Were you able to fix this?
We have similar issue.

Thanks 🙏

Nowhere I can think of right now... I'll try to poke around 🙂

SuccessfulKoala55 No logs from clearml in the console, anywhere else I can look?

I'll take a look

You say it happens at the middle of the experiment? any logs that you think might be related?

There must be some unescaping for some reason

The project name is:
"semantics_net/training"

I

ok CostlyOstrich36 , SuccessfulKoala55 :
The links differ in the very beginning -
semantics_net%252Ftrainingbecomes:
semantics_net%2FtrainingWhen I change it, I can see the image.

The last one is working or not working?

If I right click and do copy image address - I get:
http://10.156.91.102:8081/semantics_net%252Ftraining/db_lms_preds_as_inputtest.86df109a78e94a21ae000bdd4e4cae97/metrics/Images/binary%20preds/Images_binary%20preds_00000041.jpeg?X-Amz-Date=1634799860337

Whenever I restart the task, I can see images for a while, then after few iteration, I can't anymore

not working: http://10.156.91.102:8081/semantics_net%2Ftraining/db_lms_preds_as_inputtest.86df109a78e94a21ae000bdd4e4cae97/metrics/Images/binary%20preds/Images_binary%20preds_00000070.jpeg

working:
http://10.156.91.102:8080/projects/a5f62c76468a4a4c8a6cd4d8e88203e1/experiments/86df109a78e94a21ae000bdd4e4cae97/info-output/debugImages?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update

Can you share an example of a working URL and a non-working URL?

CostlyOstrich36 , The issue does come back, after I kill, restart, and wait a few cycles.
Any luck to reproduce it?

I killed and restarted the task (it created a new task, with same name, but different id) - and now I can see images again

BoredPigeon26 , it looks like the file isn't accessible through your browser. Are you sure the remote machine files are accessible?

Not Found The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

CostlyOstrich36 Page Not Found

BoredPigeon26 , when you copy and paste the link provided to you by the UI into the browser, can you see the image?

I have a local server not reachable from the internet.

tf.summary.image('binary preds', plt_c1, max_outputs=5, step=self.get_step(),
description="Left to right: inputs | sigmoid prediction | labels (blue label, white ignore)") `plt_c1 is a tf tensor .

Can you please provide a snippet of how the debug images are saved, also an example url would be useful :)

CostlyOstrich36

at some point the button of "Older Images" become grey

BoredPigeon26 , you can scroll through iterations 🙂

CostlyOstrich36
When I just started seeing this issue I could see images from previous iteration. Now some time passed and in the viewer I can't go back to the very first iterations.

BoredPigeon26 , are images from previous iterations still showing?