Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
We Are Getting "Fetch Experiment Failed" In Ui In One Of Projects. Other Projects Are Ok. In

We are getting "Fetch experiment failed" in UI in one of projects. Other projects are OK.
In apiserver.log we see:
[ERROR] [clearml.service_repo] Empty string keys not allowed without dpath.options.ALLOW_EMPTY_STRING_KEYS=True [2022-05-12 08:14:23,062] [8] [ERROR] [clearml.service_repo] Returned 500 for tasks.get_by_id_ex in 57ms, msg=Empty string keys not allowed without dpath.options.ALLOW_EMPTY_STRING_KEYS=True
Any idea what is wrong and how to fix?

  
  
Posted one year ago
Votes Newest

Answers 20


LackadaisicalHedgehong78 this is caused by some JSON structure with an empty string as a key in the request - can you get the entire request contents as sent by the WebApp? (you should be able to find it in the developer settings of the browser, under the network section)

  
  
Posted one year ago

Are you running a self deployed server? What is the version if that is the case?

  
  
Posted one year ago

self deployed server
WebApp: 1.3.0-165 • Server: 1.3.0-165 • API: 2.17
CostlyOstrich36

  
  
Posted one year ago

Yes,
When I refresh the browser window I am getting same Payload as I sent above and following is what I see in DevTools.
Also full response is:
{"meta":{"id":"670c3a1b2e994711a56e8ecb16a4c337","trx":"670c3a1b2e994711a56e8ecb16a4c337","endpoint":{"name":"tasks.get_by_id_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":500,"result_subcode":0,"result_msg":"Empty string keys not allowed without dpath.options.ALLOW_EMPTY_STRING_KEYS=True","error_stack":"Traceback (most recent call last):\n File \"/opt/clearml/apiserver/service_repo/service_repo.py\", line 286, in handle_call\n ret = endpoint.func(call, company, call.data_model)\n File \"/opt/clearml/apiserver/services/tasks.py\", line 243, in get_by_id_ex\n company=company_id, query_dict=call_data, allow_public=True,\n File \"/opt/clearml/apiserver/database/model/base.py\", line 699, in get_many_with_join\n return helper.project(results, projection_func)\n File \"/opt/clearml/apiserver/database/projection.py\", line 330, in project\n (*item, collect_ids(item[0])) for item in ref_projection.items()\n File \"/opt/clearml/apiserver/database/projection.py\", line 328, in <listcomp>\n tup\n File \"/opt/clearml/apiserver/database/projection.py\", line 330, in <genexpr>\n (*item, collect_ids(item[0])) for item in ref_projection.items()\n File \"/opt/clearml/apiserver/database/projection.py\", line 325, in collect_ids\n return list(filter(None, set(chain.from_iterable(all_ids))))\n File \"/opt/clearml/apiserver/database/projection.py\", line 323, in <genexpr>\n for res in results\n File \"/opt/clearml/apiserver/database/projection.py\", line 281, in _search\n dpath.path.paths(obj, dirs=True, skip=True)\n File \"/usr/local/lib/python3.6/site-packages/dpath/path.py\", line 103, in paths\n for child in paths(v, dirs, leaves, newpath, skip):\n File \"/usr/local/lib/python3.6/site-packages/dpath/path.py\", line 103, in paths\n for child in paths(v, dirs, leaves, newpath, skip):\n File \"/usr/local/lib/python3.6/site-packages/dpath/path.py\", line 95, in paths\n raise dpath.exceptions.InvalidKeyName(\"Empty string keys not allowed without \"\ndpath.exceptions.InvalidKeyName: Empty string keys not allowed without dpath.options.ALLOW_EMPTY_STRING_KEYS=True\n","error_data":{}},"data":{}}

  
  
Posted one year ago

{"id":["17e6b8cb533943f09512fcde813d1e39"],"only_fields":["id","name","user.name","company","type","status","status_changed","status_message","status_reason","comment","created","last_update","last_change","completed","started","parent.name","parent.project.name","project.name","output","models","models.output.model.name","models.output.model.project","models.output.model.design","models.output.model.uri","models.output.model.framework","models.output.model.created","models.output.model.task","models.output.model.task.name","models.output.model.task.project.name","models.output.model.task.user.name","models.input.model.user","models.input.model.user.name","models.input.model.project","models.input.model.name","models.input.model.created","models.input.model.labels","models.input.model.design","models.input.model.uri","models.input.model.framework","models.input.model.task","models.input.model.task.name","models.input.model.task.project","models.input.model.task.project.name","execution","hyperparams","execution.queue.name","script.binary","script.repository","script.tag","script.branch","script.version_num","script.entry_point","script.working_dir","script.requirements","system_tags","published","last_iteration","last_worker","tags","active_duration","container","runtime"]}

  
  
Posted one year ago

http://<CLEARML_URL>:8080/api/v2.17/tasks.get_by_id_ex
{"meta":{"id":"19b597cce159435694ec4f2fef17a485","trx":"19b597cce159435694ec4f2fef17a485","endpoint":{"name":"tasks.get_by_id_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":400,"result_subcode":11,"result_msg":"Missing required fields: missing=(id)","error_stack":null,"error_data":{}},"data":{}}SuccessfulKoala55

  
  
Posted one year ago

The request looks OK, I can only assume there's a task document in the DB that somehow has this in one of the fields, the issue is how to find it...

  
  
Posted one year ago

The last request you sent is the one showing as an error when the popup appears?

  
  
Posted one year ago

Oh, wait, this is not the original error, but another error

  
  
Posted one year ago

You can do "view source" and copy the JSON 🙂

  
  
Posted one year ago

That's the response - can you send the request payload?

  
  
Posted one year ago

CostlyOstrich36 SuccessfulKoala55 FYI
issue was in
task.set_parameters() https://clear.ml/docs/latest/docs/references/sdk/task/#set_parameters
When trying to reproduce our users issue we have set argument( dictionary ) with one of key s to be empty string and got the above error.

It would be great ClearML will output appropriate warning - something like " parameters issue" and not server error(500, Fetch experiment failed )

  
  
Posted one year ago

SuccessfulKoala55
We have tried to remove project and created a new one and still getting the error.
What is "suspicious" is that we see 2 projects with the same name - one as subproject and it is empty and one as regular project with experiments that can not be fetched.
Any idea?

  
  
Posted one year ago

I think this would require digging into the mongodb in order to fine the offending document, than understanding how it got there

  
  
Posted one year ago

LackadaisicalHedgehong78 , the issue doesn't seem to reproduce on my end with 1.5.0. Can you try upgrading your server?

  
  
Posted one year ago

SuccessfulKoala55 sorry for bothering you 🙂
It also fails on same when you have "/" in keys - seems related to https://pypi.org/project/dpath/ ?

  
  
Posted one year ago

LackadaisicalHedgehong78 good job finding that out 👍 - we'll make sure that's well-protected in future versions

  
  
Posted one year ago

In general keys should not contain / 🙂

  
  
Posted one year ago

image

  
  
Posted one year ago

image

  
  
Posted one year ago
639 Views
20 Answers
one year ago
13 days ago
Tags