Do any of these API calls have a "Dataset Content" field anywhere in the "configuration" section?

Do you see any errors in the mongo or elastic containers when you create a dataset?

Also, does this issue reproduce when you create datasets via the CLI rather then the SDK?

So you migrated your server and since then this issue appeared?

Hi @<1523701070390366208:profile|CostlyOstrich36> , this is what our devops engineer said:

the proxy-body-size limitation crashed for the Clearml api, for WEB and FileServer I set it to unlimited, but for the API I didn't change it.

No, i'm not seeing that "Dataset Content" section. We have some older datasets that were copied from a prior server deployment that do have the section and it appears in the UI.

What version of the server are you running? What version of the SDK also?

I'll check

Server (see screenshot). Thanks!

Hi @<1533620191232004096:profile|NuttyLobster9> , are you self hosting ClearML?

Hi @<1523701070390366208:profile|CostlyOstrich36> , update for you here. I had noticed that the issue was not present for smaller datasets, which led us to discover that the problem was being caused by some nginx (I think) settings with the new server deployment. This was blocking the upload of the "dataset content" object. So our devops team was able to resolve the issue. Thanks very much for your help.

I think this is what you're looking for but let me know if you meant something different:

{
    "meta": {
        "id": "76fffdf3b04247fa8f0c3fc0743b3ccb",
        "trx": "76fffdf3b04247fa8f0c3fc0743b3ccb",
        "endpoint": {
            "name": "tasks.get_by_id_ex",
            "requested_version": "2.30",
            "actual_version": "1.0"
        },
        "result_code": 200,
        "result_subcode": 0,
        "result_msg": "OK",
        "error_stack": "",
        "error_data": {}
    },
    "data": {
        "tasks": [
            {
                "comment": "Auto-generated at 2024-08-21 14:39:28 UTC by root@a37cd5bc31c4",
                "configuration": {
                    "Dataset Struct": {
                        "name": "Dataset Struct",
                        "value": "{\n  \"0\": {\n    \"job_id\": \"177265adca46459f8b19d7669ab5e5d5\",\n    \"status\": \"in_progress\",\n    \"last_update\": 1724251291,\n    \"parents\": [],\n    \"job_size\": 840767386,\n    \"name\": \"my-dataset\",\n    \"version\": \"1.0.0\"\n  }\n}",
                        "type": "json",
                        "description": "Structure of the dataset"
                    }
                },
                "id": "177265adca46459f8b19d7669ab5e5d5",
                "name": "my-dataset",
                "runtime": {
                    "orig_dataset_name": "my-dataset",
                    "orig_dataset_id": "177265adca46459f8b19d7669ab5e5d5",
                    "version": "1.0.0",
                    "ds_file_count": 29608,
                    "ds_link_count": 0,
                    "ds_total_size": 840767386,
                    "ds_total_size_compressed": 845212762,
                    "ds_change_add": 29608,
                    "ds_change_remove": 0,
                    "ds_change_modify": 0,
                    "ds_change_size": 840767386
                },
                "status": "completed"
            }
        ]
    }
}

Hi John, yes we are .

SDK: clearml==1.15.1

It seems so, yes. I'm not the one who did the server migration, but as a user I believe this is when I started noticing the issue for new datasets created after the migration.

I don't see any console errors

Please open developer tools (F12) and see what is returned in the network from tasks.get_by_id_ex
Also please see if there are any errors in the console

Hi @<1533620191232004096:profile|NuttyLobster9> , thank you for the update. Can you please point out what were the changes that were done?