Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, I Am Trying To Deploy Clearml Solution On Top Of Amazon Ecs With Fargate. I Am Using Efs For My Data Persistence Layer Where I Have Created Multiple Access Points For Different Components That Need To Be Deployed As Part Of The Clearml Solution. H

Hi all, I am trying to deploy ClearML solution on top of Amazon ECS with Fargate. I am using EFS for my data persistence layer where I have created multiple access points for different components that need to be deployed as part of the ClearML solution. However, I am facing difficulties with various "Permission Denied" errors such as "Permission denied: '/usr/share/nginx/html/configuration.json" or pread() "/etc/nginx/sites-enabled/clearml" failed (21: Is a directory) when launching ClearML webserver. I have tried to solve these errors for multiple files but it is always some other permission denied error or file not found. According to your docker compose sample, no volumes are being mounted for webserver container. Do you have any advice how the webserver module could be deployed on Fargate or what kind of filesystem permissions need to be set?

I managed to launch ClearML API, Fileserver and all the databases. Only the webserver part is showing these issues.

  
  
Posted 2 months ago
Votes Newest

Answers 4


@<1523701087100473344:profile|SuccessfulKoala55> , if you have any other comments, it would be much appreciated!

  
  
Posted 2 months ago

regarding running as root, I dont think I have been running as root as I had set privileged=False for the most of the runs. However, I think I tried one run with privileged=True , however again I faced the similar error...

  
  
Posted 2 months ago

Hi @<1523701087100473344:profile|SuccessfulKoala55> , I believe you are right in terms of the paths being internal to the container. It could be that the the integration between the EFS filesystem and the Fargate task is causing issues as well.

here are some of the example of the errors thrown from CloudWatch:

update_from_env.py: error: argument output_file: can't open '/usr/share/nginx/html/configuration.json': [Errno 13] Permission denied: '/usr/share/nginx/html/configuration.json'
/opt/clearml/wrapper.sh: line 82: /etc/nginx/sites-enabled/default: Permission denied
runc create failed: unable to start container process: exec: "/opt/clearml/wrapper.sh": stat /opt/clearml/wrapper.sh: no such file or directory
2024/04/01 13:49:06 [crit] 10#10: pread() "/etc/nginx/sites-enabled/clearml" failed (21: Is a directory)
update_from_env.py: error: argument output_file: can't open '/usr/share/nginx/html/configuration.json': [Errno 13] Permission denied: '/usr/share/nginx/html/configuration.json'

These logs are all from different errors that were shown on different occasions. The changes that I have been making between these runs are mounting different mount points from the EFS Access Point to the Fargate task.

Below is one example of the Fargate task definition that I have tried to run:

{
    "taskDefinitionArn": "arn:aws:ecs:eu-west-1:<account-id>:task-definition/LocalDeploymentStageClearMLWebServiceStackstorageServicetasktaskdefinition3F87B49C:11",
    "containerDefinitions": [
        {
            "name": "clearml-webserver",
            "image": "allegroai/clearml:latest",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "web",
                    "containerPort": 8080,
                    "hostPort": 8080,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "command": [
                "webserver"
            ],
            "environment": [],
            "environmentFiles": [],
            "mountPoints": [
                {
                    "sourceVolume": "webData",
                    "containerPath": "/var/log/clearml",
                    "readOnly": false
                },
                {
                    "sourceVolume": "webData",
                    "containerPath": "/usr/share/nginx/html",
                    "readOnly": false
                },
                {
                    "sourceVolume": "webData",
                    "containerPath": "/etc/nginx/sites-enabled/",
                    "readOnly": false
                },
                {
                    "sourceVolume": "webData",
                    "containerPath": "/opt/clearml",
                    "readOnly": false
                }
            ],
            "volumesFrom": [],
            "user": "1000",
            "dockerSecurityOptions": [],
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "-Local-DeploymentStage-ClearMLWebServiceStack-storageServicetasktaskdefinitionclearmlwebserverLogGroup-bQPMOGsHG7rj",
                    "awslogs-region": "eu-west-1",
                    "awslogs-stream-prefix": "/clearml/clearml-webserver"
                },
                "secretOptions": []
            },
            "systemControls": [],
            "credentialSpecs": []
        }
    ],
    "family": "LocalDeploymentStageClearMLWebServiceStackstorageServicetasktaskdefinition3F87B49C",
    "taskRoleArn": "arn:aws:iam::<account-id>:role/-Local-DeploymentStag-storageServiceTaskRole63F-DFomn9heIMJt",
    "executionRoleArn": "arn:aws:iam::<account-id>:role/-Local-DeploymentStag-storageServicetasktaskdef-oYg0bEXy3X2Z",
    "networkMode": "awsvpc",
    "revision": 11,
    "volumes": [
        {
            "name": "webData",
            "efsVolumeConfiguration": {
                "fileSystemId": "fs-123",
                "rootDirectory": "/",
                "transitEncryption": "ENABLED",
                "authorizationConfig": {
                    "accessPointId": "fsap-123",
                    "iam": "ENABLED"
                }
            }
        }
    ],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "ecs.capability.efsAuth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "ecs.capability.efs"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
        },
        {
            "name": "com.amazonaws.ecs.capability.task-iam-role"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "ecs.capability.task-eni"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "512",
    "memory": "1024",
    "registeredAt": "2024-03-31T19:29:15.512Z",
    "registeredBy": "arn:aws:sts::<account-id>:assumed-role/Admin/",
    "tags": []
}
  
  
Posted 2 months ago

Hi @<1685822572245028864:profile|LittleCow54> , these are all paths internal to the container, as far as I know - are you running the container as root?
Also, can you include a complete docker logs of these errors?

  
  
Posted 2 months ago
151 Views
4 Answers
2 months ago
2 months ago
Tags