Hi, I got some issue when using clearml-task --docker with AWS autoscaler. The error shows no space left on device , and my docker image is 12GB. I've tested other docker images and found out that it works when the docker image size is < 10GB. I think the issue can probably be solved by configuring docker daemon --storage-opt dm.basesize=20G ? But I don't know how to add this docker command in clearml, how can I add this argument to clearml-task?

2024-07-11 13:34:22
Unable to find image 'mzhengtelos/algorithms_ml:latest' locally
latest: Pulling from mzhengtelos/algorithms_ml
aece8493d397: Pulling fs layer
5e3b7ee77381: Pulling fs layer
5bd037f007fd: Pulling fs layer
4cda774ad2ec: Pulling fs layer
775f22adee62: Pulling fs layer
263fc748118f: Pulling fs layer
16c36d0187d0: Pulling fs layer
e7a56570655c: Pulling fs layer
507fc9045cba: Pulling fs layer
86e47324ec85: Pulling fs layer
a549b5b2df64: Pulling fs layer
16c36d0187d0: Waiting
507fc9045cba: Waiting
134cd1804ce0: Pulling fs layer
e7a56570655c: Waiting
7d86d9210a68: Pulling fs layer
a7057d26357e: Pulling fs layer
86e47324ec85: Waiting
c0100074397a: Pulling fs layer
0cb3dddb9ea0: Pulling fs layer
a549b5b2df64: Waiting
7d86d9210a68: Waiting
4cda774ad2ec: Waiting
0cb3dddb9ea0: Waiting
263fc748118f: Waiting
c0100074397a: Waiting
5e3b7ee77381: Download complete
aece8493d397: Verifying Checksum
aece8493d397: Download complete
4cda774ad2ec: Download complete
5bd037f007fd: Verifying Checksum
5bd037f007fd: Download complete
16c36d0187d0: Download complete
e7a56570655c: Verifying Checksum
e7a56570655c: Download complete
507fc9045cba: Verifying Checksum
507fc9045cba: Download complete
a549b5b2df64: Download complete
aece8493d397: Pull complete
5e3b7ee77381: Pull complete
134cd1804ce0: Verifying Checksum
134cd1804ce0: Download complete
5bd037f007fd: Pull complete
4cda774ad2ec: Pull complete
775f22adee62: Pull complete
7d86d9210a68: Verifying Checksum
7d86d9210a68: Download complete
a7057d26357e: Verifying Checksum
a7057d26357e: Download complete
2024-07-11 13:34:27
86e47324ec85: Verifying Checksum
86e47324ec85: Download complete
2024-07-11 13:34:32
263fc748118f: Verifying Checksum
263fc748118f: Download complete
2024-07-11 13:34:48
263fc748118f: Pull complete
16c36d0187d0: Pull complete
e7a56570655c: Pull complete
507fc9045cba: Pull complete
2024-07-11 13:34:59
86e47324ec85: Pull complete
a549b5b2df64: Pull complete
134cd1804ce0: Pull complete
c0100074397a: Verifying Checksum
c0100074397a: Download complete
7d86d9210a68: Pull complete
a7057d26357e: Pull complete
2024-07-11 13:36:35
0cb3dddb9ea0: Verifying Checksum
0cb3dddb9ea0: Download complete
2024-07-11 13:37:02
c0100074397a: Pull complete
2024-07-11 13:43:11
docker: failed to register layer: write /root/.cache/pypoetry/artifacts/c9/9b/70/1d15c007b4534b7df1ffed58d1f99cfcda825162aee8284e6148426a23/nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl: no space left on device.
See 'docker run --help'.