Here's a full description of the layout:
Remote agent + entire ClearML docker suite running on host A. Host A also has a /data/clearml folder mounted to it and to it's docker containers (I've edited the docker-compose to add this mount point) Connect to host A, use StorageManager on the /data/clearml folder for some early troubles (e.g. long .list call) Use the same connection to run a task with execute_remotely and download_folder and see it crash 😞

This was changed ~14 months ago, I guess not many people use the local file storage?

https://github.com/allegroai/clearml/commit/95105cbe6a0910ee0c37db707e32549ceaf266eb#diff-9b02810ceabbfd705225d0a7181a53b956b58479e125dd64c5f6c5c8c999e142R373

This also appears in the error log:

StorageManager.download_folder(cache_dir.as_posix(), local_folder=".") File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/manager.py", line 278, in download_folder for path in helper.list(prefix=remote_url): File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 596, in list res = self._driver.list_container_objects(self._container, ex_prefix=prefix) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 2244, in list_container_objects return list(self.iterate_container_objects(container)) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 1942, in _get_objects yield self._make_object(container, object_name) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 1902, in _make_object raise ValueError("Cannot access target path \"{}\"".format(full_path)) ValueError: Cannot access target path "/run/systemd/generator/systemd-networkd.service"

Generally the StorageManager seems a bit slow, even a simple StorageManager.list(...) on a local path seems to take a long time

Hi UnevenDolphin73 , what OS are you using? what is the clearml version?

Ubuntu 18.04, latest clearml version

The error seems to come from this line:
self._driver = _FileStorageDriver(str(path_driver_uri.root)) (line #353 in clearml/storage/helper.py
Where if the path_driver is a local path, then the _FileStorageDriver starts with a base_path = '/' , and then takes extremely long time at iterating over the entire file system (e.g. in _get_objects , line #1931 in helper.py )

about the error, do you have network manager running?

On my remote server..?

I guess not many people use the local file storage

I’m using it 🙂
How can I reproduce this issue? what should I have as cache_dir ? ~/.clearml ?

It's also sufficient to see StorageManager.list("/data/clear") takes a really long time to return no results

Indeed with ~ the .root call ends with an empty string, so it has a bit of different flow

let me try to reproduce this issue

some absolute path, i.e. /data/clearml

Can you check the running user has all the permissions?

It does (root in a docker container); it shouldn't touch /run/systemd/generator/systemd-networkd.service anyway though

UnevenDolphin73 I cant reproduce this issue on my side 😞 can you give me some hints how to?

Btw TimelyPenguin76 this should also be a good starting point:
First create the target directory and add some files:sudo mkdir /data/clearml sudo chmod 777 -R /data/clearml touch /data/clearml/foo touch /data/clearml/bar touch /data/clearml/bazThen list the files using the StorageManager. It shouldn't take more than a few miliseconds.` from clearml import StorageManager

%%timeit
StorageManager.list("/data/clearml")

-> 21.2 s ± 328 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) `

👍 let me try to reproduce with it. can you write the change you edited in the docker-compose ?

Bump?

21s is just ridiculous, it's scanning the entire file system starting at /

Any leads TimelyPenguin76 ? I've also tried setting up a minio s3 bucket, but I'm not sure if the remote agent has copied the credentials and host 🤔

Added the following line under volumes for apiserver , fileserver , agent-services :
- /data/clearml:/data/clearml