Hi @<1523701087100473344:profile|SuccessfulKoala55> , I believe you are right in terms of the paths being internal to the container. It could be that the the integration between the EFS filesystem and the Fargate task is causing issues as well.

here are some of the example of the errors thrown from CloudWatch:

update_from_env.py: error: argument output_file: can't open '/usr/share/nginx/html/configuration.json': [Errno 13] Permission denied: '/usr/share/nginx/html/configuration.json'

/opt/clearml/wrapper.sh: line 82: /etc/nginx/sites-enabled/default: Permission denied

runc create failed: unable to start container process: exec: "/opt/clearml/wrapper.sh": stat /opt/clearml/wrapper.sh: no such file or directory

2024/04/01 13:49:06 [crit] 10#10: pread() "/etc/nginx/sites-enabled/clearml" failed (21: Is a directory)

update_from_env.py: error: argument output_file: can't open '/usr/share/nginx/html/configuration.json': [Errno 13] Permission denied: '/usr/share/nginx/html/configuration.json'

These logs are all from different errors that were shown on different occasions. The changes that I have been making between these runs are mounting different mount points from the EFS Access Point to the Fargate task.

Below is one example of the Fargate task definition that I have tried to run:

{
    "taskDefinitionArn": "arn:aws:ecs:eu-west-1:<account-id>:task-definition/LocalDeploymentStageClearMLWebServiceStackstorageServicetasktaskdefinition3F87B49C:11",
    "containerDefinitions": [
        {
            "name": "clearml-webserver",
            "image": "allegroai/clearml:latest",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "web",
                    "containerPort": 8080,
                    "hostPort": 8080,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "command": [
                "webserver"
            ],
            "environment": [],
            "environmentFiles": [],
            "mountPoints": [
                {
                    "sourceVolume": "webData",
                    "containerPath": "/var/log/clearml",
                    "readOnly": false
                },
                {
                    "sourceVolume": "webData",
                    "containerPath": "/usr/share/nginx/html",
                    "readOnly": false
                },
                {
                    "sourceVolume": "webData",
                    "containerPath": "/etc/nginx/sites-enabled/",
                    "readOnly": false
                },
                {
                    "sourceVolume": "webData",
                    "containerPath": "/opt/clearml",
                    "readOnly": false
                }
            ],
            "volumesFrom": [],
            "user": "1000",
            "dockerSecurityOptions": [],
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "-Local-DeploymentStage-ClearMLWebServiceStack-storageServicetasktaskdefinitionclearmlwebserverLogGroup-bQPMOGsHG7rj",
                    "awslogs-region": "eu-west-1",
                    "awslogs-stream-prefix": "/clearml/clearml-webserver"
                },
                "secretOptions": []
            },
            "systemControls": [],
            "credentialSpecs": []
        }
    ],
    "family": "LocalDeploymentStageClearMLWebServiceStackstorageServicetasktaskdefinition3F87B49C",
    "taskRoleArn": "arn:aws:iam::<account-id>:role/-Local-DeploymentStag-storageServiceTaskRole63F-DFomn9heIMJt",
    "executionRoleArn": "arn:aws:iam::<account-id>:role/-Local-DeploymentStag-storageServicetasktaskdef-oYg0bEXy3X2Z",
    "networkMode": "awsvpc",
    "revision": 11,
    "volumes": [
        {
            "name": "webData",
            "efsVolumeConfiguration": {
                "fileSystemId": "fs-123",
                "rootDirectory": "/",
                "transitEncryption": "ENABLED",
                "authorizationConfig": {
                    "accessPointId": "fsap-123",
                    "iam": "ENABLED"
                }
            }
        }
    ],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "ecs.capability.efsAuth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "ecs.capability.efs"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
        },
        {
            "name": "com.amazonaws.ecs.capability.task-iam-role"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "ecs.capability.task-eni"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "512",
    "memory": "1024",
    "registeredAt": "2024-03-31T19:29:15.512Z",
    "registeredBy": "arn:aws:sts::<account-id>:assumed-role/Admin/",
    "tags": []
}

regarding running as root, I dont think I have been running as root as I had set privileged=False for the most of the runs. However, I think I tried one run with privileged=True , however again I faced the similar error...

@<1523701087100473344:profile|SuccessfulKoala55> , if you have any other comments, it would be much appreciated!

Hi @<1685822572245028864:profile|LittleCow54> , these are all paths internal to the container, as far as I know - are you running the container as root?
Also, can you include a complete docker logs of these errors?