Hi everone! We're trying to start using ClearML for both experiment tracking and data management, and I have a question about ClearML-Data: Is it possible to specify two different storage locations for a dataset?
Background: We're using both local machines with shared network drives and AWS EC2 instances for training. If we use local machines, I would like to get the datasets from our network drive, while when using AWS, I would like to get the data from S3. Thus I'd like to have the same data for the same dataset available in both locations. Is that possible?
Thanks for your help!
Hi @<1618418423996354560:profile|JealousMole49> , why not just use different datasets? Just to make sure I'm understanding correctly - you have a duplication of data on both s3 and local?
Yes, ideally I'd like to ensure that they are always in sync. They will be updated from time to time, adding new versions and having two separate datasets sounds like I'd always have to do this twice...
