@<1569496075083976704:profile|SweetShells3> were you able to fix this?

BoredPigeon26 SuccessfulKoala55 CostlyOstrich36 Hi
Were you able to fix this?
We have similar issue.

CostlyOstrich36 SuccessfulKoala55 FYI
issue was in
task.set_parameters() https://clear.ml/docs/latest/docs/references/sdk/task/#set_parameters
When trying to reproduce our users issue we have set argument( dictionary ) with one of key s to be empty string and got the above error.

It would be great ClearML will output appropriate warning - something like " parameters issue" and not server error(500, Fetch experiment failed )

SuccessfulKoala55 sorry for bothering you 🙂
It also fails on same when you have "/" in keys - seems related to https://pypi.org/project/dpath/ ?

self deployed server
WebApp: 1.3.0-165 • Server: 1.3.0-165 • API: 2.17
CostlyOstrich36

SuccessfulKoala55
We have tried to remove project and created a new one and still getting the error.
What is "suspicious" is that we see 2 projects with the same name - one as subproject and it is empty and one as regular project with experiments that can not be fetched.
Any idea?

CostlyOstrich36
NO errors in developer tools and result code is 200:
` {"meta":{"id":"f131cde7b77545a5b4802e73f1b5e78e","trx":"f131cde7b77545a5b4802e73f1b5e78e","endpoint":{"name":"tasks.get_all_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":"","error_data":{}},"data":{"tasks":[{"company":{"id":"d1bd92a3b039400cbafc60a7a5b1e52b"},"id":"4b2fcf54203e4930b7a9a7b511e31ca3","last_change":"2022-11-21T20:40:53.107000+00:00...

We tried with copying on test machine and it worked(delete and then restore tasks in DB - it appears again in UI).
When did same on prod - nothing happened.
Also we see all data in the fileserver is not removed and mongo shows 17000 tasks so it looks like tasks were removed from UI but still appear in Mongo DB and locally at file system.
CostlyOstrich36
` > db.task.count()
17262

db.task__trash.count()
383
db.task__trash__trash.count()
15
db.task__trash.aggregate([ {$merge: "task"}...

FYI CostlyOstrich36
After CLearML restart, all experiments appear again 😃

HI SoreFlamingo44 ,
We are getting same error?
How did you fix this?

` {"id":["17e6b8cb533943f09512fcde813d1e39"],"only_fields":["id","name","user.name","company","type","status","status_changed","status_message","status_reason","comment","created","last_update","last_change","completed","started","parent.name","parent.project.name","project.name","output","models","models.output.model.name","models.output.model.project","models.output.model.design","models.output.model.uri","models.output.model.framework","models.output.model.created","models.output.model.tas...

Hi CostlyOstrich36
We indeed see tasks in task__trash collection in mongodb backend database.
Is there any way to restore it?

Also can we see in logs who triggered the command?

Thanks CostlyOstrich36
Actually I was able to find IP of the machine where API call was triggered in web logs and found the user who run the delete action.
User tried to remove only archived experiments in his project( tried several times and got some errors ) and that is what we see in API call - somehow Clearml removed all server experiments 🤔
Any idea why this might happened if user only run "delete archived experiments of his project" in WEB UI ?
` xx.xxx.xxx.xx - - [21/Nov/2022:17:3...

Yes,
When I refresh the browser window I am getting same Payload as I sent above and following is what I see in DevTools.
Also full response is:
` {"meta":{"id":"670c3a1b2e994711a56e8ecb16a4c337","trx":"670c3a1b2e994711a56e8ecb16a4c337","endpoint":{"name":"tasks.get_by_id_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":500,"result_subcode":0,"result_msg":"Empty string keys not allowed without dpath.options.ALLOW_EMPTY_STRING_KEYS=True","error_stack":"Traceback (most recen...

http://<CLEARML_URL>:8080/api/v2.17/tasks.get_by_id_ex
{"meta":{"id":"19b597cce159435694ec4f2fef17a485","trx":"19b597cce159435694ec4f2fef17a485","endpoint":{"name":"tasks.get_by_id_ex","requested_version":"2.17","actual_version":"1.0"},"result_code":400,"result_subcode":11,"result_msg":"Missing required fields: missing=(id)","error_stack":null,"error_data":{}},"data":{}}SuccessfulKoala55

@<1523701842515595264:profile|PleasantOwl46> were you able to fix this?
@<1523701087100473344:profile|SuccessfulKoala55> what might be a fix for this?