Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
When Using

When using StorageManager.download_folder , I get the following error:
Traceback (most recent call last): File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 1900, in _make_object stat = os.stat(full_path) FileNotFoundError: [Errno 2] No such file or directory: '/run/systemd/generator/systemd-networkd.service'

  
  
Posted one year ago
Votes Newest

Answers 23


Here's a full description of the layout:
Remote agent + entire ClearML docker suite running on host A. Host A also has a /data/clearml folder mounted to it and to it's docker containers (I've edited the docker-compose to add this mount point) Connect to host A, use StorageManager on the /data/clearml folder for some early troubles (e.g. long .list call) Use the same connection to run a task with execute_remotely and download_folder and see it crash šŸ˜ž

  
  
Posted one year ago

This also appears in the error log:

StorageManager.download_folder(cache_dir.as_posix(), local_folder=".") File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/manager.py", line 278, in download_folder for path in helper.list(prefix=remote_url): File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 596, in list res = self._driver.list_container_objects(self._container, ex_prefix=prefix) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 2244, in list_container_objects return list(self.iterate_container_objects(container)) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 1942, in _get_objects yield self._make_object(container, object_name) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 1902, in _make_object raise ValueError("Cannot access target path \"{}\"".format(full_path)) ValueError: Cannot access target path "/run/systemd/generator/systemd-networkd.service"

  
  
Posted one year ago

Generally the StorageManager seems a bit slow, even a simple StorageManager.list(...) on a local path seems to take a long time

  
  
Posted one year ago

Hi UnevenDolphin73 , what OS are you using? what is the clearml version?

  
  
Posted one year ago

Ubuntu 18.04, latest clearml version

  
  
Posted one year ago

The error seems to come from this line:
self._driver = _FileStorageDriver(str(path_driver_uri.root)) (line #353 in clearml/storage/helper.py
Where if the path_driver is a local path, then the _FileStorageDriver starts with a base_path = '/' , and then takes extremely long time at iterating over the entire file system (e.g. in _get_objects , line #1931 in helper.py )

  
  
Posted one year ago

about the error, do you have network manager running?

  
  
Posted one year ago

On my remote server..?

  
  
Posted one year ago

I guess not many people use the local file storage

Iā€™m using it šŸ™‚
How can I reproduce this issue? what should I have as cache_dir ? ~/.clearml ?

  
  
Posted one year ago

It's also sufficient to see StorageManager.list("/data/clear") takes a really long time to return no results

  
  
Posted one year ago

Indeed with ~ the .root call ends with an empty string, so it has a bit of different flow

  
  
Posted one year ago

let me try to reproduce this issue

  
  
Posted one year ago

some absolute path, i.e. /data/clearml

  
  
Posted one year ago

Can you check the running user has all the permissions?

  
  
Posted one year ago

It does (root in a docker container); it shouldn't touch /run/systemd/generator/systemd-networkd.service anyway though

  
  
Posted one year ago

UnevenDolphin73 I cant reproduce this issue on my side šŸ˜ž can you give me some hints how to?

  
  
Posted one year ago

Btw TimelyPenguin76 this should also be a good starting point:
First create the target directory and add some files:sudo mkdir /data/clearml sudo chmod 777 -R /data/clearml touch /data/clearml/foo touch /data/clearml/bar touch /data/clearml/bazThen list the files using the StorageManager. It shouldn't take more than a few miliseconds.` from clearml import StorageManager

%%timeit
StorageManager.list("/data/clearml")

-> 21.2 s Ā± 328 ms per loop (mean Ā± std. dev. of 7 runs, 1 loop each) `

  
  
Posted one year ago

šŸ‘ let me try to reproduce with it. can you write the change you edited in the docker-compose ?

  
  
Posted one year ago

Bump?

  
  
Posted one year ago

21s is just ridiculous, it's scanning the entire file system starting at /

  
  
Posted one year ago

Any leads TimelyPenguin76 ? I've also tried setting up a minio s3 bucket, but I'm not sure if the remote agent has copied the credentials and host šŸ¤”

  
  
Posted one year ago

Added the following line under volumes for apiserver , fileserver , agent-services :
- /data/clearml:/data/clearml

  
  
Posted one year ago
107 Views
23 Answers
one year ago
4 months ago
Tags
Similar posts