The "new problem" was not being able to view the console, scalars, plots, debug samples of previous experiments (probably because the reference to them was in /usr/share/elasticsearch/data , which I changed to /var/lib/elasticsearch/data in the new installation, in an attempt to install it without having sudo permissions).

@<1523701070390366208:profile|CostlyOstrich36> thanks for the reply.
So, what I need to do is simply copy the data directory (with the fileserver, redis, elastic_7 and mongo_4 subdirectories) to the new server to /opt/clearml/data?
Suppose I don't do the mongo+ES migration immediately, should I still be able to view the list of experiments and things like scalars, plots, configuration etc for each experiment?

@<1523703097560403968:profile|CumbersomeCormorant74> Hi, thanks for the suggestion, but unfortunately, it did not work.

The value is the same.
As I mentioned before, the server version I'm working with does not have the async_delete container. Unfortunately, due to internal considerations, the version update will not take place in the near future, so I am having the system admin delete them for me manually every once in a while.

@<1523701070390366208:profile|CostlyOstrich36> It works! thanks!

Great. Thanks!

Is there a way to view the version in the web GUI?

This container may have not been introduced in this version yet, I don't see one in docker ps

Do you think that updating to a new version will probably fix this?

WebApp: 1.9.1-312 • Server: 1.9.1-312 • API: 2.23

As a temporary solution, shutting down the entire docker-compose, deleting the left over files using administrator permissions and then bringing it back up again, does this sound reasonable?

There is no chance of corrupting other experiments or databases?

I meant, as a temporary solution instead of upgrading

Keeping the current version and deleting manually

None
Thanks!

SDK 1.14.4
Server 1.14.1-451

Oh, so the task has an internal keepalive mechanism and me calling time.sleep() for more than 2 hours prevents it from working?

OK thanks. Just curious then, suppose you use the task for normal experiment tracking, you do Task.init() in the beginning as usual and train you model and your epochs are longer then 2 hours and you only print/report stuff at epoch end, would this cause the task to abort too?

After some experimenting it seems that the situation improves when I call task.mark_started(force=True) before each task.upload_artifact() instead of just once in the beginning of the script.

Seems there are two approaches, either "revive" before each upload, or somehow keep it always "Running", do you have an idea how the second approach can be achieved? (I did not call task.close() or task.mark_*() anywhere).

@<1576381444509405184:profile|ManiacalLizard2> , thanks, that was my initial solution, but I had some trouble with reusing the previously created task for the scheduler when the process that made the call to TaskScheduler.add_task() was interrupted.