Hi, I'm trying to understand if clearml supports my usecase: i generate my own data (problem-specific) and generation process is uploading a Parquet dataset (in fact, parquet contains multiple files obv.) to S3. Is there a way to "register" the dataset in Clearml without doing local copy (dataset is > 300GB)?
Thanks, didnt check here for a while, i have managed to find out about this myself but Thx anyway:)
Hi GreasyWalrus57 , sorry but didn’t get that.
You want to register the data? you can do it with clearml-data and then use this task to connect between tasks and data
