Thanks for the reponse.
I have try this exactly and still the same issue 😕 . I think that the clearml-agent adds a specific search for the pytorch wheels withe ther PytorchRequirements class and then raises this issue .
I don't quite understand why clearml tries to solve specificaly for the pytorch package 🤔 , You could just add the repos url in pip extra and let it solve the wheels for itself ?

Hello, I am a data engineer but new to clearml.
If you train in batches then you should only get acces to the batch of document in those 100k. You could use s3 and implement the fetch in the get_item method :)

Here is the full log of the experiment