Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
When Using

When using StorageManager.download_folder , I get the following error:
Traceback (most recent call last): File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 1900, in _make_object stat = os.stat(full_path) FileNotFoundError: [Errno 2] No such file or directory: '/run/systemd/generator/systemd-networkd.service'

  
  
Posted 2 years ago
Votes Newest

Answers 23


Any leads TimelyPenguin76 ? I've also tried setting up a minio s3 bucket, but I'm not sure if the remote agent has copied the credentials and host šŸ¤”

  
  
Posted 2 years ago

This also appears in the error log:

StorageManager.download_folder(cache_dir.as_posix(), local_folder=".") File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/manager.py", line 278, in download_folder for path in helper.list(prefix=remote_url): File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 596, in list res = self._driver.list_container_objects(self._container, ex_prefix=prefix) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 2244, in list_container_objects return list(self.iterate_container_objects(container)) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 1942, in _get_objects yield self._make_object(container, object_name) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 1902, in _make_object raise ValueError("Cannot access target path \"{}\"".format(full_path)) ValueError: Cannot access target path "/run/systemd/generator/systemd-networkd.service"

  
  
Posted 2 years ago

The error seems to come from this line:
self._driver = _FileStorageDriver(str(path_driver_uri.root)) (line #353 in clearml/storage/helper.py
Where if the path_driver is a local path, then the _FileStorageDriver starts with a base_path = '/' , and then takes extremely long time at iterating over the entire file system (e.g. in _get_objects , line #1931 in helper.py )

  
  
Posted 2 years ago

Generally the StorageManager seems a bit slow, even a simple StorageManager.list(...) on a local path seems to take a long time

  
  
Posted 2 years ago

about the error, do you have network manager running?

  
  
Posted 2 years ago

On my remote server..?

  
  
Posted 2 years ago

Hi UnevenDolphin73 , what OS are you using? what is the clearml version?

  
  
Posted 2 years ago

Ubuntu 18.04, latest clearml version

  
  
Posted 2 years ago

I guess not many people use the local file storage

Iā€™m using it šŸ™‚
How can I reproduce this issue? what should I have as cache_dir ? ~/.clearml ?

  
  
Posted 2 years ago

some absolute path, i.e. /data/clearml

  
  
Posted 2 years ago

It's also sufficient to see StorageManager.list("/data/clear") takes a really long time to return no results

  
  
Posted 2 years ago

Indeed with ~ the .root call ends with an empty string, so it has a bit of different flow

  
  
Posted 2 years ago

Can you check the running user has all the permissions?

  
  
Posted 2 years ago

It does (root in a docker container); it shouldn't touch /run/systemd/generator/systemd-networkd.service anyway though

  
  
Posted 2 years ago

let me try to reproduce this issue

  
  
Posted 2 years ago

Bump?

  
  
Posted 2 years ago

UnevenDolphin73 I cant reproduce this issue on my side šŸ˜ž can you give me some hints how to?

  
  
Posted 2 years ago

Btw TimelyPenguin76 this should also be a good starting point:
First create the target directory and add some files:sudo mkdir /data/clearml sudo chmod 777 -R /data/clearml touch /data/clearml/foo touch /data/clearml/bar touch /data/clearml/bazThen list the files using the StorageManager. It shouldn't take more than a few miliseconds.` from clearml import StorageManager

%%timeit
StorageManager.list("/data/clearml")

-> 21.2 s Ā± 328 ms per loop (mean Ā± std. dev. of 7 runs, 1 loop each) `

  
  
Posted 2 years ago

šŸ‘ let me try to reproduce with it. can you write the change you edited in the docker-compose ?

  
  
Posted 2 years ago

Added the following line under volumes for apiserver , fileserver , agent-services :
- /data/clearml:/data/clearml

  
  
Posted 2 years ago

21s is just ridiculous, it's scanning the entire file system starting at /

  
  
Posted 2 years ago

Here's a full description of the layout:
Remote agent + entire ClearML docker suite running on host A. Host A also has a /data/clearml folder mounted to it and to it's docker containers (I've edited the docker-compose to add this mount point) Connect to host A, use StorageManager on the /data/clearml folder for some early troubles (e.g. long .list call) Use the same connection to run a task with execute_remotely and download_folder and see it crash šŸ˜ž

  
  
Posted 2 years ago
500 Views
23 Answers
2 years ago
one year ago
Tags
Similar posts