Hey all, I've been doing some hyperparameter tuning today using a clearML managed dataset. However, I kept running into the following error: Error loading dataset: pickle data was truncated .
After a bit of digging I found out this is likely due to the fact that all instances appear to be writing to the same file in the .cache folder, namely, '/home/usr/.clearml/cache/storage_manager/datasets/ds_dataset_id' . When multiple processes run at the same time some processes are still writing to this file while others are already trying to fetch it, causing the error. This seemed like a pretty common issue to me and I thought there would probably be a workaround for this but I am unable to find it. Could anyone here advice me on how to deal with this issue?
We have a server that has many agents running on it because there are many instances where training can be run over several agents as a single agent doesn't take up all the resources available to the server.
Hi LargeHamster21 , are you running multiple instances of the agent on the same machine? If that is the case, can you elaborate on the use case?
