I recently migrated my clearml server to a different machine. i copied all the data folder as recommended above. on the new clearml server i can see all my old experiments and datasets. unfortunately, when running a task with a dataset from the previous machine, the tasks fails and writes the old server ip.
2023-03-12 12:55:59,934 - clearml.storage - ERROR - Could not download None .............
i replaced everywhere i could find the old ip with the new one.
i tried searching for the old ip in the data folder that was copied. it seems that the ip exists in the elastic, mongo and redis dirs in binaries files.
also when i go to the clearml server -> datasets -> any data that was copied -> task information -> details -> output -> destination, i see the old ip
is there a setting that i can define to overwrite it?
maybe something like the sdk.storage.path_substitution?
I did the move as directed by @<1523701070390366208:profile|CostlyOstrich36> ,
We have our new clearml server filled with data from old server.
however, the new agents are not able to pull the tasks. All tasks remain pending.
Please let us know what could be the cause of this
What do you see in the agent logs? Are there any errors? Can you verify they are indeed working with the new server?
It's possible that the agents aren't configured to the new server?
Hi @<1523701523954012160:profile|ShallowCormorant89> , I think you can simply spin down all the containers and copy everything in
that is amazing! thank you @<1523701070390366208:profile|CostlyOstrich36>
we did delete the old configs and
clearml-agent init 'ed everything, pointing to the new IP