Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
HomelyBluewhale47
Moderator
5 Questions, 35 Answers
  Active since 07 June 2024
  Last activity 7 months ago

Reputation

0

Badges 1

27 × Eureka!
0 Votes
4 Answers
640 Views
0 Votes 4 Answers 640 Views
Hi Everyone! Is There A Way To Store Env_Vars Or Secrets In Clearml When We Use Remote Agent To Run The Task? Actually I Have Too Many Envs That Are Confidential.
Hi Everyone! Is there a way to store ENV_VARs or secrets in clearML when we use remote agent to run the task? Actually I have too many envs that are confiden...
7 months ago
0 Votes
1 Answers
366 Views
0 Votes 1 Answers 366 Views
Hello Everyone. Is There A Way To Ignore Editing
Hello everyone. Is there a way to ignore editing clearml.conf file and set ENV vars for aws s3 credentials while launching clearml-agent? In my usecase I sta...
aws
5 months ago
0 Votes
4 Answers
562 Views
0 Votes 4 Answers 562 Views
Clearml-Agent==1.8.1
clearml-agent==1.8.1 I already have a config file. And when I pass --config-file using below command it throws error. Command: clearml-agent daemon --queue d...
7 months ago
0 Votes
0 Answers
566 Views
0 Votes 0 Answers 566 Views
Hi Everyone, I’M Reporting A Confusion Matrix Using Below Code.
Hi Everyone, I’m reporting a confusion matrix using below code. clearml_logger.report_confusion_matrix( title="Confusion Matrix (Accuracy)", series="ignored"...
7 months ago
0 Votes
33 Answers
34K Views
0 Votes 33 Answers 34K Views
Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:
Hi everyone. I’m struggling to setup minio storage. Below is what I’m adding in my credentials and when I try to create a new dataset using below command; I ...
aws
7 months ago
0 Clearml-Agent==1.8.1

image

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

Removed port and set secure as true.

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

image

7 months ago
0 Clearml-Agent==1.8.1

Thanks @<1523701087100473344:profile|SuccessfulKoala55>

7 months ago
0 Hello, Im Having Huge Performance Issues On Large Clearml Datasets How Can I Link To Parent Dataset Without Parent Dataset Files. I Want To Create A Smaller Subset Of Parent Dataset, Like 5% Of It. To Achieve This, I Have To Call Remove_Files() To 60K It

Hi @<1523701435869433856:profile|SmugDolphin23>
This is follow up question from my side.

Is it efficient to use clearML data for hundred of thousands of image dataset. I’ve a concern about performance.

So basically I’m training a transformers classifier. which need at least 300-400K images.

7 months ago
0 Hello, Im Having Huge Performance Issues On Large Clearml Datasets How Can I Link To Parent Dataset Without Parent Dataset Files. I Want To Create A Smaller Subset Of Parent Dataset, Like 5% Of It. To Achieve This, I Have To Call Remove_Files() To 60K It

Thanks a lot @<1523701435869433856:profile|SmugDolphin23>
will go through this one.
your quick reply really means a lot. Thanks again!!

7 months ago
0 Hello, Im Having Huge Performance Issues On Large Clearml Datasets How Can I Link To Parent Dataset Without Parent Dataset Files. I Want To Create A Smaller Subset Of Parent Dataset, Like 5% Of It. To Achieve This, I Have To Call Remove_Files() To 60K It

And just FYI: one dataset size can beyond 80GB.

7 months ago
0 Hello, Im Having Huge Performance Issues On Large Clearml Datasets How Can I Link To Parent Dataset Without Parent Dataset Files. I Want To Create A Smaller Subset Of Parent Dataset, Like 5% Of It. To Achieve This, I Have To Call Remove_Files() To 60K It

Well I can understand well your problem. @<1590514584836378624:profile|AmiableSeaturtle81>
Please let me know if you have answers for my problem. 😄

7 months ago
0 Hello, Im Having Huge Performance Issues On Large Clearml Datasets How Can I Link To Parent Dataset Without Parent Dataset Files. I Want To Create A Smaller Subset Of Parent Dataset, Like 5% Of It. To Achieve This, I Have To Call Remove_Files() To 60K It

@<1590514584836378624:profile|AmiableSeaturtle81>
Yes, I can also feel that slowness.

But what about if we have millions of pdfs and we need to generate a image dataset along with their bounding boxes. We cannot generate it on the fly while training the model as it will very very very slow.
So I’m doing all preprocessing the upload preprocessed data back to storage.
And while training again pulling preprocessed dataset from storage. 😄

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

I also tried passing the default port for minio for host in config
eg: host: “ostore.xyz.tech:9000”

But getting same error.

7 months ago
0 Hi Everyone! Is There A Way To Store Env_Vars Or Secrets In Clearml When We Use Remote Agent To Run The Task? Actually I Have Too Many Envs That Are Confidential.

Hi @<1523701070390366208:profile|CostlyOstrich36>
FYI: I’m not using docker agent.

I set the env vars as you mentioned above in config. And I can see the those while starting agent using command: clearml-agent daemon --queue default

While running below code locally eg: python test.py I’m loading the env vars that is working fine.
But my question is when we clone the same task from clearml UI and send it to default queue; So can below piece of code can use those envs?
I tried sendi...

7 months ago
0 Hello, Im Having Huge Performance Issues On Large Clearml Datasets How Can I Link To Parent Dataset Without Parent Dataset Files. I Want To Create A Smaller Subset Of Parent Dataset, Like 5% Of It. To Achieve This, I Have To Call Remove_Files() To 60K It

And what I feel; pulling data by using some client eg: minio-client for minio is faster than clearml

7 months ago
0 Hello, Im Having Huge Performance Issues On Large Clearml Datasets How Can I Link To Parent Dataset Without Parent Dataset Files. I Want To Create A Smaller Subset Of Parent Dataset, Like 5% Of It. To Achieve This, I Have To Call Remove_Files() To 60K It

Yes, this is what I’m doing currently. But strugginling to manage the versions and all.

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

raise ValueError(“Insufficient permissions (delete failed) for {}“.format(base_url))
ValueError: Insufficient permissions (delete failed) for None

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

host: None

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

Hi @<1523701070390366208:profile|CostlyOstrich36>
After removing the port number from command given by you.
Re: your message
with the combination of None :port/bucket for --storage ?

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

Also tried changing files_server in config but didn’t work
image

7 months ago
0 Clearml-Agent==1.8.1

Am I missing anything? because I can clearly see the usage of --config-file using clearml-agent --help

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

FYI: I also changed the files_server in config

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

Thanks a lot @<1523701070390366208:profile|CostlyOstrich36> emoji people:simple_smile 🙏

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

Again throwing Insufficient permissions.

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

I passed only None

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

yes

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

Am I missing anything in config?

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

Yup, tried that too.

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

Yes, I tried with default port of minio (9000). didn’t work either.

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

still got same error

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

sure

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: “ None ”

7 months ago
0 Hi Everyone. I’M Struggling To Setup Minio Storage. Below Is What I’M Adding In My Credentials And When I Try To Create A New Dataset Using Below Command; I Get Errors: Configs:

it is picking http, it should https?

7 months ago
Show more results compactanswers