CostlyOstrich36 SuccessfulKoala55 FYI
issue was in
task.set_parameters() https://clear.ml/docs/latest/docs/references/sdk/task/#set_parameters
When trying to reproduce our users issue we have set argument( dictionary ) with one of key s to be empty string and got the above error.

It would be great ClearML will output appropriate warning - something like " parameters issue" and not server error(500, Fetch experiment failed )

SuccessfulKoala55 sorry for bothering you 🙂
It also fails on same when you have "/" in keys - seems related to https://pypi.org/project/dpath/ ?

Are you running a self deployed server? What is the version if that is the case?

self deployed server
WebApp: 1.3.0-165 • Server: 1.3.0-165 • API: 2.17
CostlyOstrich36

The request looks OK, I can only assume there's a task document in the DB that somehow has this in one of the fields, the issue is how to find it...

LackadaisicalHedgehong78 good job finding that out 👍 - we'll make sure that's well-protected in future versions

Oh, wait, this is not the original error, but another error

You can do "view source" and copy the JSON 🙂

LackadaisicalHedgehong78 , the issue doesn't seem to reproduce on my end with 1.5.0. Can you try upgrading your server?

SuccessfulKoala55
We have tried to remove project and created a new one and still getting the error.
What is "suspicious" is that we see 2 projects with the same name - one as subproject and it is empty and one as regular project with experiments that can not be fetched.
Any idea?

That's the response - can you send the request payload?

In general keys should not contain / 🙂

LackadaisicalHedgehong78 this is caused by some JSON structure with an empty string as a key in the request - can you get the entire request contents as sent by the WebApp? (you should be able to find it in the developer settings of the browser, under the network section)

{"id":["17e6b8cb533943f09512fcde813d1e39"],"only_fields":["id","name","user.name","company","type","status","status_changed","status_message","status_reason","comment","created","last_update","last_change","completed","started","parent.name","parent.project.name","project.name","output","models","models.output.model.name","models.output.model.project","models.output.model.design","models.output.model.uri","models.output.model.framework","models.output.model.created","models.output.model.task","models.output.model.task.name","models.output.model.task.project.name","models.output.model.task.user.name","models.input.model.user","models.input.model.user.name","models.input.model.project","models.input.model.name","models.input.model.created","models.input.model.labels","models.input.model.design","models.input.model.uri","models.input.model.framework","models.input.model.task","models.input.model.task.name","models.input.model.task.project","models.input.model.task.project.name","execution","hyperparams","execution.queue.name","script.binary","script.repository","script.tag","script.branch","script.version_num","script.entry_point","script.working_dir","script.requirements","system_tags","published","last_iteration","last_worker","tags","active_duration","container","runtime"]}

The last request you sent is the one showing as an error when the popup appears?

Yes,
When I refresh the browser window I am getting same Payload as I sent above and following is what I see in DevTools.
Also full response is:
{"meta":{"id":"670c3a1b2e994711a56e8ecb16a4c337","trx":"670c3a1b2e994711a56e8ecb16a4c337","endpoint":{"name":"tasks.get_by_id_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":500,"result_subcode":0,"result_msg":"Empty string keys not allowed without dpath.options.ALLOW_EMPTY_STRING_KEYS=True","error_stack":"Traceback (most recent call last):\n File \"/opt/clearml/apiserver/service_repo/service_repo.py\", line 286, in handle_call\n ret = endpoint.func(call, company, call.data_model)\n File \"/opt/clearml/apiserver/services/tasks.py\", line 243, in get_by_id_ex\n company=company_id, query_dict=call_data, allow_public=True,\n File \"/opt/clearml/apiserver/database/model/base.py\", line 699, in get_many_with_join\n return helper.project(results, projection_func)\n File \"/opt/clearml/apiserver/database/projection.py\", line 330, in project\n (*item, collect_ids(item[0])) for item in ref_projection.items()\n File \"/opt/clearml/apiserver/database/projection.py\", line 328, in <listcomp>\n tup\n File \"/opt/clearml/apiserver/database/projection.py\", line 330, in <genexpr>\n (*item, collect_ids(item[0])) for item in ref_projection.items()\n File \"/opt/clearml/apiserver/database/projection.py\", line 325, in collect_ids\n return list(filter(None, set(chain.from_iterable(all_ids))))\n File \"/opt/clearml/apiserver/database/projection.py\", line 323, in <genexpr>\n for res in results\n File \"/opt/clearml/apiserver/database/projection.py\", line 281, in _search\n dpath.path.paths(obj, dirs=True, skip=True)\n File \"/usr/local/lib/python3.6/site-packages/dpath/path.py\", line 103, in paths\n for child in paths(v, dirs, leaves, newpath, skip):\n File \"/usr/local/lib/python3.6/site-packages/dpath/path.py\", line 103, in paths\n for child in paths(v, dirs, leaves, newpath, skip):\n File \"/usr/local/lib/python3.6/site-packages/dpath/path.py\", line 95, in paths\n raise dpath.exceptions.InvalidKeyName(\"Empty string keys not allowed without \"\ndpath.exceptions.InvalidKeyName: Empty string keys not allowed without dpath.options.ALLOW_EMPTY_STRING_KEYS=True\n","error_data":{}},"data":{}}

http://<CLEARML_URL>:8080/api/v2.17/tasks.get_by_id_ex
{"meta":{"id":"19b597cce159435694ec4f2fef17a485","trx":"19b597cce159435694ec4f2fef17a485","endpoint":{"name":"tasks.get_by_id_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":400,"result_subcode":11,"result_msg":"Missing required fields: missing=(id)","error_stack":null,"error_data":{}},"data":{}}SuccessfulKoala55

I think this would require digging into the mongodb in order to fine the offending document, than understanding how it got there