Hi @<1523701987055505408:profile|WittyOwl57> ,
is the plot data stored in mongo, or does mongo just store some links?
Plot data is stored in the ElasticSearch database. I am not sure how is this related to the rest of your question as they pertain to images 🙂
If I, say, copy the clearml data directory from the existing machine to a different location, and copy the tensorboard data to the same absolute path, will that work?
That should work. I'm not sure why tansorboard is mentioned here, but if you're talking about the fileserver storage folder, than it would work.
In the future I want to avoid this problem (of having the move experiment tracking, merge experiments from different machines, etc). What is the best practice for that? I would try to store everything (including actual experiment data, e.g. tensorboard, logs, etc) to blob storage, but I don't think that is possible.
Scaling should not be an issue. Using blob storage for uploaded artifacts and images is part of the system's design (by using the files_server
and output_uri
configuration options on the client side). Anything else can be handles by scaling up the databases, which as you mentioned should not be an issue.