The "new problem" was not being able to view the console, scalars, plots, debug samples of previous experiments (probably because the reference to them was in /usr/share/elasticsearch/data
, which I changed to /var/lib/elasticsearch/data
in the new installation, in an attempt to install it without having sudo permissions).
Hi, I have upgraded my ClearML server from version 1.9.1-312 to version 1.13.0-414.
After the upgrade much of outputs of the previously existing experiments is not showing in the web UI.
Specifically, the console, scalars, plots, debug samples tabs are completely empty (even though the files that correspond to the debug samples are still physically in the fileserver directory).
The execution, configuration, artifacts and info tabs seem to be OK.
Any idea how to fix this?
PS. I have tried retrieving the debug samples using the Task.get_debug_samples() method (with new SDK version 1.13.2), it works for new experiments, but not for those created before the upgrade.
UPDATE : seems the cause was an edit I made to the docker-compose.yml, I changed the data directory of elastic search from /usr/share/elasticsearch/data
to /var/lib/elasticsearch/data
following an answer from SO after I experienced permission errors on the original directory within the elasticsearch container. It solved the permissions error, but created a new problem 😅