LackadaisicalHedgehong78 this is caused by some JSON structure with an empty string as a key in the request - can you get the entire request contents as sent by the WebApp? (you should be able to find it in the developer settings of the browser, under the network section)

CostlyOstrich36 SuccessfulKoala55 FYI
issue was in
task.set_parameters() https://clear.ml/docs/latest/docs/references/sdk/task/#set_parameters
When trying to reproduce our users issue we have set argument( dictionary ) with one of key s to be empty string and got the above error.

It would be great ClearML will output appropriate warning - something like " parameters issue" and not server error(500, Fetch experiment failed )

You can do "view source" and copy the JSON 🙂

The request looks OK, I can only assume there's a task document in the DB that somehow has this in one of the fields, the issue is how to find it...

In general keys should not contain / 🙂

Are you running a self deployed server? What is the version if that is the case?

I think this would require digging into the mongodb in order to fine the offending document, than understanding how it got there

SuccessfulKoala55
We have tried to remove project and created a new one and still getting the error.
What is "suspicious" is that we see 2 projects with the same name - one as subproject and it is empty and one as regular project with experiments that can not be fetched.
Any idea?

LackadaisicalHedgehong78 good job finding that out 👍 - we'll make sure that's well-protected in future versions

Oh, wait, this is not the original error, but another error

self deployed server
WebApp: 1.3.0-165 • Server: 1.3.0-165 • API: 2.17
CostlyOstrich36

LackadaisicalHedgehong78 , the issue doesn't seem to reproduce on my end with 1.5.0. Can you try upgrading your server?

That's the response - can you send the request payload?

SuccessfulKoala55 sorry for bothering you 🙂
It also fails on same when you have "/" in keys - seems related to https://pypi.org/project/dpath/ ?

Yes,
When I refresh the browser window I am getting same Payload as I sent above and following is what I see in DevTools.
Also full response is:
{"meta":{"id":"670c3a1b2e994711a56e8ecb16a4c337","trx":"670c3a1b2e994711a56e8ecb16a4c337","endpoint":{"name":"tasks.get_by_id_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":500,"result_subcode":0,"result_msg":"Empty string keys not allowed without dpath.options.ALLOW_EMPTY_STRING_KEYS=True","error_stack":"Traceback (most recent call last):\n File \"/opt/clearml/apiserver/service_repo/service_repo.py\", line 286, in handle_call\n ret = endpoint.func(call, company, call.data_model)\n File \"/opt/clearml/apiserver/services/tasks.py\", line 243, in get_by_id_ex\n company=company_id, query_dict=call_data, allow_public=True,\n File \"/opt/clearml/apiserver/database/model/base.py\", line 699, in get_many_with_join\n return helper.project(results, projection_func)\n File \"/opt/clearml/apiserver/database/projection.py\", line 330, in project\n (*item, collect_ids(item[0])) for item in ref_projection.items()\n File \"/opt/clearml/apiserver/database/projection.py\", line 328, in <listcomp>\n tup\n File \"/opt/clearml/apiserver/database/projection.py\", line 330, in <genexpr>\n (*item, collect_ids(item[0])) for item in ref_projection.items()\n File \"/opt/clearml/apiserver/database/projection.py\", line 325, in collect_ids\n return list(filter(None, set(chain.from_iterable(all_ids))))\n File \"/opt/clearml/apiserver/database/projection.py\", line 323, in <genexpr>\n for res in results\n File \"/opt/clearml/apiserver/database/projection.py\", line 281, in _search\n dpath.path.paths(obj, dirs=True, skip=True)\n File \"/usr/local/lib/python3.6/site-packages/dpath/path.py\", line 103, in paths\n for child in paths(v, dirs, leaves, newpath, skip):\n File \"/usr/local/lib/python3.6/site-packages/dpath/path.py\", line 103, in paths\n for child in paths(v, dirs, leaves, newpath, skip):\n File \"/usr/local/lib/python3.6/site-packages/dpath/path.py\", line 95, in paths\n raise dpath.exceptions.InvalidKeyName(\"Empty string keys not allowed without \"\ndpath.exceptions.InvalidKeyName: Empty string keys not allowed without dpath.options.ALLOW_EMPTY_STRING_KEYS=True\n","error_data":{}},"data":{}}

The last request you sent is the one showing as an error when the popup appears?

{"id":["17e6b8cb533943f09512fcde813d1e39"],"only_fields":["id","name","user.name","company","type","status","status_changed","status_message","status_reason","comment","created","last_update","last_change","completed","started","parent.name","parent.project.name","project.name","output","models","models.output.model.name","models.output.model.project","models.output.model.design","models.output.model.uri","models.output.model.framework","models.output.model.created","models.output.model.task","models.output.model.task.name","models.output.model.task.project.name","models.output.model.task.user.name","models.input.model.user","models.input.model.user.name","models.input.model.project","models.input.model.name","models.input.model.created","models.input.model.labels","models.input.model.design","models.input.model.uri","models.input.model.framework","models.input.model.task","models.input.model.task.name","models.input.model.task.project","models.input.model.task.project.name","execution","hyperparams","execution.queue.name","script.binary","script.repository","script.tag","script.branch","script.version_num","script.entry_point","script.working_dir","script.requirements","system_tags","published","last_iteration","last_worker","tags","active_duration","container","runtime"]}

http://<CLEARML_URL>:8080/api/v2.17/tasks.get_by_id_ex
{"meta":{"id":"19b597cce159435694ec4f2fef17a485","trx":"19b597cce159435694ec4f2fef17a485","endpoint":{"name":"tasks.get_by_id_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":400,"result_subcode":11,"result_msg":"Missing required fields: missing=(id)","error_stack":null,"error_data":{}},"data":{}}SuccessfulKoala55