Also tried changing files_server in config but didn’t work

raise ValueError(“Insufficient permissions (delete failed) for {}“.format(base_url))
ValueError: Insufficient permissions (delete failed) for None

I passed only None

Thanks a lot @<1523701070390366208:profile|CostlyOstrich36> 🙏

it is picking http, it should https?

Yup, tried that too.

clearml-data create --storage "

" --project dummyproject --name testingds1

clearml-data - Dataset Management & Versioning CLI
Creating a new dataset:

Error: Insufficient permissions (write failed) for ostore.xyz.tech

Removed port and set secure as true.

Am I missing anything in config?

still got same error

Everytime I start training; first download dataset using minio client and then do further operations.

Can we integrate minio easily with clearML?

@<1590514584836378624:profile|AmiableSeaturtle81>
Yes, I can also feel that slowness.

But what about if we have millions of pdfs and we need to generate a image dataset along with their bounding boxes. We cannot generate it on the fly while training the model as it will very very very slow.
So I’m doing all preprocessing the upload preprocessed data back to storage.
And while training again pulling preprocessed dataset from storage. 😄

Hi @<1523701435869433856:profile|SmugDolphin23>
This is follow up question from my side.

Is it efficient to use clearML data for hundred of thousands of image dataset. I’ve a concern about performance.

So basically I’m training a transformers classifier. which need at least 300-400K images.

Thanks a lot @<1523701435869433856:profile|SmugDolphin23>
will go through this one.
your quick reply really means a lot. Thanks again!!

And just FYI: one dataset size can beyond 80GB.

Well I can understand well your problem. @<1590514584836378624:profile|AmiableSeaturtle81>
Please let me know if you have answers for my problem. 😄

And what I feel; pulling data by using some client eg: minio-client for minio is faster than clearml

Hi @<1523701070390366208:profile|CostlyOstrich36>
FYI: I’m not using docker agent.

I set the env vars as you mentioned above in config. And I can see the those while starting agent using command: clearml-agent daemon --queue default

While running below code locally eg: python test.py I’m loading the env vars that is working fine.
But my question is when we clone the same task from clearml UI and send it to default queue; So can below piece of code can use those envs?
I tried sendi...

Thanks @<1523701087100473344:profile|SuccessfulKoala55>

Yes, this is what I’m doing currently. But strugginling to manage the versions and all.

For below config and command it worked
command: clearml-data create --storage " None " --project dummyproject --name testingds2

Though I’m able to connect using minio client with the same creds without any issues.

Again throwing Insufficient permissions.

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: “ None ”

FYI: I also changed the files_server in config

Hi John
Well, I didn’t setup anything extra 🙂

Am I missing anything? because I can clearly see the usage of --config-file using clearml-agent --help

Hi @<1523701070390366208:profile|CostlyOstrich36>
After removing the port number from command given by you.
Re: your message
with the combination of None :port/bucket for --storage ?

Basically we want to avoid having .conf file and it would be great in automation.