Can you share an example of a working URL and a non-working URL?

CostlyOstrich36
When I just started seeing this issue I could see images from previous iteration. Now some time passed and in the viewer I can't go back to the very first iterations.

Can you please provide a snippet of how the debug images are saved, also an example url would be useful :)

BoredPigeon26 , are images from previous iterations still showing?

BoredPigeon26 , you can scroll through iterations 🙂

at some point the button of "Older Images" become grey

CostlyOstrich36

tf.summary.image('binary preds', plt_c1, max_outputs=5, step=self.get_step(),
description="Left to right: inputs | sigmoid prediction | labels (blue label, white ignore)") `plt_c1 is a tf tensor .

Not Found The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

BoredPigeon26 , it looks like the file isn't accessible through your browser. Are you sure the remote machine files are accessible?

BoredPigeon26 , when you copy and paste the link provided to you by the UI into the browser, can you see the image?

I have a local server not reachable from the internet.

CostlyOstrich36 Page Not Found

working:
http://10.156.91.102:8080/projects/a5f62c76468a4a4c8a6cd4d8e88203e1/experiments/86df109a78e94a21ae000bdd4e4cae97/info-output/debugImages?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-last_update

If I right click and do copy image address - I get:
http://10.156.91.102:8081/semantics_net%252Ftraining/db_lms_preds_as_inputtest.86df109a78e94a21ae000bdd4e4cae97/metrics/Images/binary%20preds/Images_binary%20preds_00000041.jpeg?X-Amz-Date=1634799860337

not working: http://10.156.91.102:8081/semantics_net%2Ftraining/db_lms_preds_as_inputtest.86df109a78e94a21ae000bdd4e4cae97/metrics/Images/binary%20preds/Images_binary%20preds_00000070.jpeg

I killed and restarted the task (it created a new task, with same name, but different id) - and now I can see images again

CostlyOstrich36 , The issue does come back, after I kill, restart, and wait a few cycles.
Any luck to reproduce it?

Whenever I restart the task, I can see images for a while, then after few iteration, I can't anymore

ok CostlyOstrich36 , SuccessfulKoala55 :
The links differ in the very beginning -
semantics_net%252Ftrainingbecomes:
semantics_net%2FtrainingWhen I change it, I can see the image.

You say it happens at the middle of the experiment? any logs that you think might be related?

The project name is:
"semantics_net/training"

There must be some unescaping for some reason

The last one is working or not working?

I

BoredPigeon26 SuccessfulKoala55 CostlyOstrich36 Hi
Were you able to fix this?
We have similar issue.

SuccessfulKoala55 No logs from clearml in the console, anywhere else I can look?

Nowhere I can think of right now... I'll try to poke around 🙂

I'll take a look

Thanks 🙏