Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
LackadaisicalHedgehong78
Moderator
3 Questions, 18 Answers
  Active since 10 January 2023
  Last activity 2 months ago

Reputation

0

Badges 1

14 × Eureka!
0 Votes
20 Answers
748 Views
0 Votes 20 Answers 748 Views
We are getting "Fetch experiment failed" in UI in one of projects. Other projects are OK. In apiserver.log we see: [ERROR] [clearml.service_repo] Empty strin...
2 years ago
0 Votes
1 Answers
522 Views
0 Votes 1 Answers 522 Views
Hello, We are getting following timeout errors during the task run: 2023-08-10 13:53:36,361 - clearml.Metrics - ERROR - Action failed <500/100: events.add_ba...
10 months ago
0 Votes
10 Answers
635 Views
0 Votes 10 Answers 635 Views
one year ago
0 We Are Getting "Fetch Experiment Failed" In Ui In One Of Projects. Other Projects Are Ok. In

CostlyOstrich36 SuccessfulKoala55 FYI
issue was in
task.set_parameters() https://clear.ml/docs/latest/docs/references/sdk/task/#set_parameters
When trying to reproduce our users issue we have set argument( dictionary ) with one of key s to be empty string and got the above error.

It would be great ClearML will output appropriate warning - something like " parameters issue" and not server error(500, Fetch experiment failed )

2 years ago
0 We Are Getting "Fetch Experiment Failed" In Ui In One Of Projects. Other Projects Are Ok. In

SuccessfulKoala55 sorry for bothering you 🙂
It also fails on same when you have "/" in keys - seems related to https://pypi.org/project/dpath/ ?

2 years ago
0 We Are Getting "Fetch Experiment Failed" In Ui In One Of Projects. Other Projects Are Ok. In

http://<CLEARML_URL>:8080/api/v2.17/tasks.get_by_id_ex
{"meta":{"id":"19b597cce159435694ec4f2fef17a485","trx":"19b597cce159435694ec4f2fef17a485","endpoint":{"name":"tasks.get_by_id_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":400,"result_subcode":11,"result_msg":"Missing required fields: missing=(id)","error_stack":null,"error_data":{}},"data":{}}SuccessfulKoala55

2 years ago
0 We Are Getting "Fetch Experiment Failed" In Ui In One Of Projects. Other Projects Are Ok. In

Yes,
When I refresh the browser window I am getting same Payload as I sent above and following is what I see in DevTools.
Also full response is:
` {"meta":{"id":"670c3a1b2e994711a56e8ecb16a4c337","trx":"670c3a1b2e994711a56e8ecb16a4c337","endpoint":{"name":"tasks.get_by_id_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":500,"result_subcode":0,"result_msg":"Empty string keys not allowed without dpath.options.ALLOW_EMPTY_STRING_KEYS=True","error_stack":"Traceback (most recen...

2 years ago
0 We Are Getting "Fetch Experiment Failed" In Ui In One Of Projects. Other Projects Are Ok. In

self deployed server
WebApp: 1.3.0-165 • Server: 1.3.0-165 • API: 2.17
CostlyOstrich36

2 years ago
0 We Are Getting "Fetch Experiment Failed" In Ui In One Of Projects. Other Projects Are Ok. In

SuccessfulKoala55
We have tried to remove project and created a new one and still getting the error.
What is "suspicious" is that we see 2 projects with the same name - one as subproject and it is empty and one as regular project with experiments that can not be fetched.
Any idea?

2 years ago
0 Hello I”M Getting A ‘Fetch Experiments Failed’ On The Ui. How Can One Solve This

HI SoreFlamingo44 ,
We are getting same error?
How did you fix this?

2 years ago
0 Hey All I See This Error Repeating Constantly, And My Metrics Are Not Updated, What Does It Mean?

@<1523701842515595264:profile|PleasantOwl46> were you able to fix this?
@<1523701087100473344:profile|SuccessfulKoala55> what might be a fix for this?

2 months ago
0 Hi, We Have Clearml Version Webapp: 1.3.0-165 • Server: 1.3.0-165 • Api: 2.17 Suddenly All Experiments Were Removed. After Checking Logs We See That Before 6 Days Elasticsearch Container Was Terminated Due To Out Of Memory And New One Was Created Instead

Thanks CostlyOstrich36
Actually I was able to find IP of the machine where API call was triggered in web logs and found the user who run the delete action.
User tried to remove only archived experiments in his project( tried several times and got some errors ) and that is what we see in API call - somehow Clearml removed all server experiments 🤔
Any idea why this might happened if user only run "delete archived experiments of his project" in WEB UI ?
` xx.xxx.xxx.xx - - [21/Nov/2022:17:3...

one year ago
0 Hi, We Have Clearml Version Webapp: 1.3.0-165 • Server: 1.3.0-165 • Api: 2.17 Suddenly All Experiments Were Removed. After Checking Logs We See That Before 6 Days Elasticsearch Container Was Terminated Due To Out Of Memory And New One Was Created Instead

We tried with copying on test machine and it worked(delete and then restore tasks in DB - it appears again in UI).
When did same on prod - nothing happened.
Also we see all data in the fileserver is not removed and mongo shows 17000 tasks so it looks like tasks were removed from UI but still appear in Mongo DB and locally at file system.
CostlyOstrich36
` > db.task.count()
17262

db.task__trash.count()
383
db.task__trash__trash.count()
15
db.task__trash.aggregate([ {$merge: "task"}...

one year ago
0 Hi, We Have Clearml Version Webapp: 1.3.0-165 • Server: 1.3.0-165 • Api: 2.17 Suddenly All Experiments Were Removed. After Checking Logs We See That Before 6 Days Elasticsearch Container Was Terminated Due To Out Of Memory And New One Was Created Instead

Hi CostlyOstrich36
We indeed see tasks in task__trash collection in mongodb backend database.
Is there any way to restore it?

Also can we see in logs who triggered the command?

one year ago
0 Hi, We Have Clearml Version Webapp: 1.3.0-165 • Server: 1.3.0-165 • Api: 2.17 Suddenly All Experiments Were Removed. After Checking Logs We See That Before 6 Days Elasticsearch Container Was Terminated Due To Out Of Memory And New One Was Created Instead

CostlyOstrich36
NO errors in developer tools and result code is 200:
` {"meta":{"id":"f131cde7b77545a5b4802e73f1b5e78e","trx":"f131cde7b77545a5b4802e73f1b5e78e","endpoint":{"name":"tasks.get_all_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":"","error_data":{}},"data":{"tasks":[{"company":{"id":"d1bd92a3b039400cbafc60a7a5b1e52b"},"id":"4b2fcf54203e4930b7a9a7b511e31ca3","last_change":"2022-11-21T20:40:53.107000+00:00...

one year ago
0 We Are Getting "Fetch Experiment Failed" In Ui In One Of Projects. Other Projects Are Ok. In

` {"id":["17e6b8cb533943f09512fcde813d1e39"],"only_fields":["id","name","user.name","company","type","status","status_changed","status_message","status_reason","comment","created","last_update","last_change","completed","started","parent.name","parent.project.name","project.name","output","models","models.output.model.name","models.output.model.project","models.output.model.design","models.output.model.uri","models.output.model.framework","models.output.model.created","models.output.model.tas...

2 years ago
0 Hi All! I Can'T Use Scalar Tab In All Experiments Due To Elastic Search Error:

@<1569496075083976704:profile|SweetShells3> were you able to fix this?

2 months ago