Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, Is

Hi,

Is clearml-task --docker able to take docker image > 10GB?

I got some issue when using clearml-task --docker with AWS autoscaler. The error shows no space left on device , and my docker image is 12GB. I've tested other docker images and found out that it works when the docker image size is < 10GB.

I've tried to run the docker pull manually in AWS EC2 and it could be downloaded successfully. So I think the issue can probably be solved by configuring docker daemon --storage-opt dm.basesize=20G ? But I don't know how to add this docker command in clearml, how can I add this argument to clearml-task?

Is there a way to have the docker extra arguments takes dockerd command instead of docker run ? I tried dockerd --storage-opt dm.basesize=20G in docker extra arugments, but it failed.

2024-07-11 13:34:22
Unable to find image 'mzhengtelos/algorithms_ml:latest' locally
latest: Pulling from mzhengtelos/algorithms_ml
aece8493d397: Pulling fs layer
5e3b7ee77381: Pulling fs layer
5bd037f007fd: Pulling fs layer
4cda774ad2ec: Pulling fs layer
775f22adee62: Pulling fs layer
263fc748118f: Pulling fs layer
16c36d0187d0: Pulling fs layer
e7a56570655c: Pulling fs layer
507fc9045cba: Pulling fs layer
86e47324ec85: Pulling fs layer
a549b5b2df64: Pulling fs layer
16c36d0187d0: Waiting
507fc9045cba: Waiting
134cd1804ce0: Pulling fs layer
e7a56570655c: Waiting
7d86d9210a68: Pulling fs layer
a7057d26357e: Pulling fs layer
86e47324ec85: Waiting
c0100074397a: Pulling fs layer
0cb3dddb9ea0: Pulling fs layer
a549b5b2df64: Waiting
7d86d9210a68: Waiting
4cda774ad2ec: Waiting
0cb3dddb9ea0: Waiting
263fc748118f: Waiting
c0100074397a: Waiting
5e3b7ee77381: Download complete
aece8493d397: Verifying Checksum
aece8493d397: Download complete
4cda774ad2ec: Download complete
5bd037f007fd: Verifying Checksum
5bd037f007fd: Download complete
16c36d0187d0: Download complete
e7a56570655c: Verifying Checksum
e7a56570655c: Download complete
507fc9045cba: Verifying Checksum
507fc9045cba: Download complete
a549b5b2df64: Download complete
aece8493d397: Pull complete
5e3b7ee77381: Pull complete
134cd1804ce0: Verifying Checksum
134cd1804ce0: Download complete
5bd037f007fd: Pull complete
4cda774ad2ec: Pull complete
775f22adee62: Pull complete
7d86d9210a68: Verifying Checksum
7d86d9210a68: Download complete
a7057d26357e: Verifying Checksum
a7057d26357e: Download complete
2024-07-11 13:34:27
86e47324ec85: Verifying Checksum
86e47324ec85: Download complete
2024-07-11 13:34:32
263fc748118f: Verifying Checksum
263fc748118f: Download complete
2024-07-11 13:34:48
263fc748118f: Pull complete
16c36d0187d0: Pull complete
e7a56570655c: Pull complete
507fc9045cba: Pull complete
2024-07-11 13:34:59
86e47324ec85: Pull complete
a549b5b2df64: Pull complete
134cd1804ce0: Pull complete
c0100074397a: Verifying Checksum
c0100074397a: Download complete
7d86d9210a68: Pull complete
a7057d26357e: Pull complete
2024-07-11 13:36:35
0cb3dddb9ea0: Verifying Checksum
0cb3dddb9ea0: Download complete
2024-07-11 13:37:02
c0100074397a: Pull complete
2024-07-11 13:43:11
docker: failed to register layer: write /root/.cache/pypoetry/artifacts/c9/9b/70/1d15c007b4534b7df1ffed58d1f99cfcda825162aee8284e6148426a23/nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl: no space left on device.
See 'docker run --help'.
  
  
Posted 4 months ago
pic
More
EnchantingPenguin77
0 × 1
Report
Votes Newest

Answers 3

In your case, you should simply preconfigure the docker service on the machine where you're running the clearml-agent

  
  
Posted 4 months ago
pic
More
SuccessfulKoala55
0 × 1
Report

Hi @<1597762318140182528:profile|EnchantingPenguin77> , this has nothing to do with clearml-task, the docker image is pulled on the machine running your clearml agent (using the docker run command), and from the log you've sent it seems the operating system there does not have enough disk space to pull the image

  
  
Posted 4 months ago
pic
More
SuccessfulKoala55
0 × 1
Report

Hi @<1523701087100473344:profile|SuccessfulKoala55> , what preconfiguration is needed for the docker service to make? I've tried to run the docker pull manually in AWS EC2 with the same docker image without the space limit issue.

  
  
Posted 4 months ago
pic
More
EnchantingPenguin77
0 × 1
Report
Write your answer
392 Views
3 Answers
4 months ago
4 months ago
Tags
clearml