CostlyOstrich36

0 Questions, 3782 Answers

Active since 10 January 2023

Last activity one year ago

Reputation

Answers 3782

0 I Am Using Clearml Pro And Pretty Regularly I Will Restart An Experiment And Nothing Will Get Logged To Clearml. It Shows The Experiment Running (For Days) And It'S Running Fine On The Pc But No Scalers Or Debug Samples Are Shown. How Do We Troubleshoot T

Console logs

4 months ago

0 Is There Any Way To: Within The Ui, Select And Compare The Scalars For More Than 10 Experiments? I'D Like To Do Something Like:

SmallDeer34 Hi! 🙂

I'm afraid that currently we support only up to 10 experiments in comparison at once. However you can add/remove experiments mid comparison. Is there a specific reason why you'd need to compare specifically 20?

Regarding the coloring, if you do 5/5 it would be supported.

3 years ago

0 Is There Any Way To: Within The Ui, Select And Compare The Scalars For More Than 10 Experiments? I'D Like To Do Something Like:

SmallDeer34 , great, thanks for the info 🙂

3 years ago

0 Is There Any Way To: Within The Ui, Select And Compare The Scalars For More Than 10 Experiments? I'D Like To Do Something Like:

SmallDeer34 , as a workaround for the issue of the graph maximize you can simply rename some of the experiments and this should fix the coloring issue for now until it is fixed in the next version 🙂

3 years ago

0 Is There Any Way To: Within The Ui, Select And Compare The Scalars For More Than 10 Experiments? I'D Like To Do Something Like:

SmallDeer34 , and they still have the same colors when you maximize the graph?

3 years ago

0 Hi! We'Re Trying To Orchestrate An Experiment Through The Clearml Ui With A Remote Agent On K8S. Our Training Job Requires A Package Only Available On Our Internal Pypi Server. How Should We Go About This Because The Job Keeps Failing With:

Hi CostlyFox64 ,

Can you try configuring your ~/clearml.conf with the following?
agent.package_manager.extra_index_url= [ "https://<USER>:<PASSWORD>@packages.<HOSTNAME>/<REPO_PATH>" ]

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

From what I understand, by default the ES has a low disk waterkmark set at 95% of the disk capacity. Once reached the shard is transitioned to a read only mode. Since you have a large disk of 1.8Tb the remaining 85Gb is below the 5%.

Basically you need to set the following env vars in elasticsearch service in the docker compose:
` - cluster.routing.allocation.disk.watermark.low=10gb
- cluster.routing.allocation.disk.watermark.high=10gb
- cluster.routing.allocation.disk.wate...

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

docker-compose.yml file you used to set up the server

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

How did you install your clearml server?

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Hi David,

What version of ClearML server & SDK are you using?

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

What happens if you use the settings I pasted?

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

In the web UI you can click the settings icon at the top right -> settings. At that screen the version should be shown at the bottom right

2 years ago

0 Hi! I’M Running An Experiment As Follows:

Can you add a small code snippet that reproduces this?

2 years ago

0 Hello, I'M Following The Tutorial Of

ScaryBluewhale66 , Hi 🙂

You would need to install ClearML-Agent to run it

2 years ago

0 Hello, I'M Following The Tutorial Of

Can you provide more of the log before the error as well?

2 years ago

0 Hi! I’M Running An Experiment As Follows:

Also, what version of clearml & clearml-agent are you using?

2 years ago

0 I'M Getting Upload Errors When Uploading Artifacts (Using Gcs) But It Uploads For Datasets Works Just Fine.

Hi @<1575656665519230976:profile|SkinnyBat30> , what version of ClearML are you using? Are you uploading datasets from the same machine also to GCS?

one year ago

0 Hello ! I Am Using The Pipelinecontroller To Orchestrate A Set Of Job. The Clearm-Agent (Workers) Are Hosted On Spot Instance (Gcp) And The Pipellinecontroller Does Not Mark The Task As Failure Whenever A Worker Gets Preempted. Is There A Way To Mark The

Is the pipelinecontroller also working on preemptible instances?

one year ago

0 We Are Hosting Our Own Clearml Server, And I Would Like Everybody In Our Team To Configure Their Clearml With

Hi SoreHorse95 , about enforcing a setting globally I think this option only exists in the Scale/Enterprise versions.

Regarding the config snippet - do you manually change it to port 80? I'm not sure how to configure this, maybe SuccessfulKoala55 might have some more insight.

one year ago

0 Hey, We'Re Seeing A Lot Of Issues With Our Clearml Self-Hosted Server These Days; It Seems Like The Api Times Out While Talking To Elasticsearch:

Hi GiganticMole91 , what version of ClearML server are you using?
Also, can you take a look inside the elastic container to see if there are any errors there?

2 years ago

0 Hey, We'Re Seeing A Lot Of Issues With Our Clearml Self-Hosted Server These Days; It Seems Like The Api Times Out While Talking To Elasticsearch:

Googling ES error: es_rejected_execution_exception and it seems that it is caused due to the excessive load on ES. Apparently the hardware cannot keep up with the pace that you're sending events batches. I would recommend working with smaller batches and checking whether the error goes away.

2 years ago

0 Hey, We'Re Seeing A Lot Of Issues With Our Clearml Self-Hosted Server These Days; It Seems Like The Api Times Out While Talking To Elasticsearch:

Were there any changes to your Elastic or your server in the past few days?

2 years ago

0 Hi, I Am Trying To Figure Out How To Get Cloud Storage Access Working In The Agent. I Am Running The Agent Locally In Docker Mode. I Set Up Gcp Storage In The Clearml.Json But It Seems Not To Get Passed To The Agent. Also Tried To Add Agent.Extra_Docker_A

Or should I set agent.google.storage {}?

Did you follow the instructions in the docs?

10 months ago

Hi @<1671689442621919232:profile|ItchyDuck87> , did you manage to register directly via the SDK?

10 months ago

0 Hey, How Can I Find Which Tasks/Projects Have The Most Metrics?

Are you self hosted or using the community?

9 months ago

0 Hey, How Can I Find Which Tasks/Projects Have The Most Metrics?

Hi @<1533257411639382016:profile|RobustRat47> , what would you define as most metrics?

9 months ago

0 Hi Guys, I'M Loving Clearml So Far Now We Can Quickly Train And Deploy Customer Models

RobustRat47 , you can optimize the connection process by using auto_connect_frameworks to select the frameworks you're using. That can speed up the process

3 years ago

0 Trying To Add Someone To My Workspace And We Keep Getting A Cannot Resolve Link Destination

Oh, can you please do the same with developer tools when user tries to accept?

one year ago

0 Trying To Add Someone To My Workspace And We Keep Getting A Cannot Resolve Link Destination

Can you please open developer tools (F12) and see what is returned in network when you try to do this?

one year ago

0 Trying To Add Someone To My Workspace And We Keep Getting A Cannot Resolve Link Destination

Strange, maybe @<1523703436166565888:profile|DeterminedCrab71> might have an idea

one year ago

Show more results