Yes, that's correct. I don't want to re-download datasets because of their large size.

SuccessfulKoala55
I initialized the task with Python:

task = Task.init(project_name=args.project_name, task_name=args.task_name)

and downloaded set of datasets later in the code:

for dataset_name in datasets_list:
clearml_dataset = clearml.Dataset.get(dataset_project=dataset_project, dataset_name=dataset_name)
clearml_dataset_path = clearml_dataset.get_local_copy()

Then I go through the resulting directories in search of the files I need, and send their pa...

Yes AgitatedDove14 , I mean multithreading. I did not quite understand your question about single Dataset version. Could you clarify the question for me, please?

At least from the logs I see in my terminal I assume that right now downloading works as on the scheme 2. This one:

Thank you for your response. It works. I want to run several workers simultaneously on the same GPU, because I have to train several, relatively simple and small, neural networks. It would be faster to train several of them at the same time on the same GPU, rather than do it consequently.

Yes, I would like to run several agents on the same GPU. I use command python -m clearml_agent daemon --queue default queue_name --docker --gpus 0 --detached

Here is the schemes of discussed variants (1. download process before ClearML 1.5.0; 2. current version; 3. proposed method). I hope they will be helpful.

Could you please point me to the piece of ClearML code related to the downloading process?

Sorry. I probably misunderstood you. I just downloaded the clearml-agent package to my machine and ran the agent with the following command: python -m clearml_agent daemon --queue default dinara --docker --detached

Hi DepressedFish57 AgitatedDove14 AnxiousSeal95 ! It's me again. I created a https://github.com/allegroai/clearml/pull/713 where I slightly changed the dataset loading code, ran some tests to estimate dataset loading times using the current and proposed approaches and tried to explain in detail the problem I see.

Or, we can download chunks in parallel, like we do it right now, but we have to prioritize the download of the earlier chunks to make extraction and downloading run in parallel.

Hi AnxiousSeal95 ! Thank you for the parallel download feature you have added. We have tested it with ClearML 1.5.0 and it seems that it is not really helpful. For a big dataset the time it takes to download does not really changes with the new feature. We indeed can download several chunks in parallel, but it turns out that while N workers are downloading N chunks, downloading speed for each worker is N times less than for a consequent download. Since that most of the chunks have the same s...