@<1523701070390366208:profile|CostlyOstrich36> It works! thanks!

None
Thanks!

SDK 1.14.4
Server 1.14.1-451

The "new problem" was not being able to view the console, scalars, plots, debug samples of previous experiments (probably because the reference to them was in /usr/share/elasticsearch/data , which I changed to /var/lib/elasticsearch/data in the new installation, in an attempt to install it without having sudo permissions).

This container may have not been introduced in this version yet, I don't see one in docker ps

@<1523703097560403968:profile|CumbersomeCormorant74> Hi, thanks for the suggestion, but unfortunately, it did not work.

Do you think that updating to a new version will probably fix this?

Is there a way to view the version in the web GUI?

WebApp: 1.9.1-312 • Server: 1.9.1-312 • API: 2.23

I meant, as a temporary solution instead of upgrading

Keeping the current version and deleting manually

As a temporary solution, shutting down the entire docker-compose, deleting the left over files using administrator permissions and then bringing it back up again, does this sound reasonable?

There is no chance of corrupting other experiments or databases?

After some experimenting it seems that the situation improves when I call task.mark_started(force=True) before each task.upload_artifact() instead of just once in the beginning of the script.

Seems there are two approaches, either "revive" before each upload, or somehow keep it always "Running", do you have an idea how the second approach can be achieved? (I did not call task.close() or task.mark_*() anywhere).

OK thanks. Just curious then, suppose you use the task for normal experiment tracking, you do Task.init() in the beginning as usual and train you model and your epochs are longer then 2 hours and you only print/report stuff at epoch end, would this cause the task to abort too?

@<1576381444509405184:profile|ManiacalLizard2> , thanks, that was my initial solution, but I had some trouble with reusing the previously created task for the scheduler when the process that made the call to TaskScheduler.add_task() was interrupted.

Oh, so the task has an internal keepalive mechanism and me calling time.sleep() for more than 2 hours prevents it from working?

The value is the same.
As I mentioned before, the server version I'm working with does not have the async_delete container. Unfortunately, due to internal considerations, the version update will not take place in the near future, so I am having the system admin delete them for me manually every once in a while.

Great. Thanks!