RattyFish27

1 Question, 15 Answers

Active since 10 January 2023

Last activity one year ago

Reputation

Badges 1

15 × Eureka!

Questions 1
Answers 15

0 Votes

27 Answers

943 Views

0 Votes 27 Answers 943 Views

Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Hi all, I’m running experiments using ClearML. The training is very slow, and I’m getting the following errors and warnings: clearml.Task - ERROR - Action fa...

clearml

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Yes I run the command from the server that host the container.
In the environment section I have:
CLEARML_ELASTIC_SERVICE_PORT: 9200
In the apiserver section I have the following:
ports: - "8008:8008"
In the fileserver section I have :
ports: - "8081:8081"

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Sure I will do it tomorrow
Thanks for the help in the meantime

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

I found it

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Not sure - I’m not the one who installed it.

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

I’m not sure where it is,
I tried to run the following:
` docker container inspect clearml-elastic | grep compose
"com.docker.compose.config-hash": "5563c3436de5f0c5e3349da7f0511ea7a8020ce4ce5dc22d161f0561e973fecd",
"com.docker.compose.container-number": "1",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "clearml",
"com.docker.compose.service": "elasticsearch",
"com.docker.com...

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Done,
I got the following outputs:
` health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .geoip_databases Mshu2LugQ2aQYmFTB8Lckg 1 0 44 74 76.8mb 76.8mb
green open events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b 7qIRfRjNRke6GuHZzRKsuw 1 0 71382012 565576 8.3gb ...

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Thanks - it works :)
It happen for the second time in the last couple of months,
Do you have any suggestions regarding why it’s happening and how to make sure it won’t happen again ?

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

I tried it (including docker down and up)
Currently giving the same errors as follows:

2022-08-15 16:15:19,676 - clearml.Task - ERROR - Action failed <500/100: tasks.reset/v1.0 (General data error (TransportError(503, 'search_phase_execution_exception')))> (force=False, clear_all=False, task=f535ef2c45cf4baaaf13c8f2fe2ac19a)
` 2022-08-15 16:16:19,845 - clearml.log - WARNING - failed logging task to backend (1 lines, <500/100: events.add_batch/v1.0 (General data error: err=('1 document(s)...

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Hi Evgeny,
I tried to run the curl command, it gave me the following error:
curl: (7) Failed to connect to localhost port 9200: Connection refused

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

But the values are smaller then 10 gb (it is 500 mb):
cluster.routing.allocation.disk.watermark.low: 500mb cluster.routing.allocation.disk.watermark.high: 500mb cluster.routing.allocation.disk.watermark.flood_stage: 500mb

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Which means the experiments will be deleted right ?
And how should I delete the index ?

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

I’m trying to find the compose file,
Any suggestions ?

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

WebApp: 1.5.0-192 • Server: 1.5.0-192 • API: 2.18

2 years ago

0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Ok Thanks