Btw TimelyPenguin76 this should also be a good starting point:
First create the target directory and add some files:sudo mkdir /data/clearml sudo chmod 777 -R /data/clearml touch /data/clearml/foo touch /data/clearml/bar touch /data/clearml/bazThen list the files using the StorageManager. It shouldn't take more than a few miliseconds.` from clearml import StorageManager
%%timeit
StorageManager.list("/data/clearml")
-> 21.2 s Ā± 328 ms per loop (mean Ā± std. dev. of 7 runs, 1 loop each) `
let me try to reproduce this issue
This was changed ~14 months ago, I guess not many people use the local file storage?
I guess not many people use the local file storage
Iām using it š
How can I reproduce this issue? what should I have as cache_dir ? ~/.clearml ?
It does (root in a docker container); it shouldn't touch /run/systemd/generator/systemd-networkd.service anyway though
This also appears in the error log:
StorageManager.download_folder(cache_dir.as_posix(), local_folder=".") File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/manager.py", line 278, in download_folder for path in helper.list(prefix=remote_url): File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 596, in list res = self._driver.list_container_objects(self._container, ex_prefix=prefix) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 2244, in list_container_objects return list(self.iterate_container_objects(container)) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 1942, in _get_objects yield self._make_object(container, object_name) File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 1902, in _make_object raise ValueError("Cannot access target path \"{}\"".format(full_path)) ValueError: Cannot access target path "/run/systemd/generator/systemd-networkd.service"
Indeed with ~ the .root call ends with an empty string, so it has a bit of different flow
It's also sufficient to see StorageManager.list("/data/clear") takes a really long time to return no results
Generally the StorageManager seems a bit slow, even a simple StorageManager.list(...) on a local path seems to take a long time
Added the following line under volumes for apiserver , fileserver , agent-services :- /data/clearml:/data/clearml
about the error, do you have network manager running?
The error seems to come from this line:self._driver = _FileStorageDriver(str(path_driver_uri.root)) (line #353 in clearml/storage/helper.py
Where if the path_driver is a local path, then the _FileStorageDriver starts with a base_path = '/' , and then takes extremely long time at iterating over the entire file system (e.g. in _get_objects , line #1931 in helper.py )
Can you check the running user has all the permissions?
Here's a full description of the layout:
Remote agent + entire ClearML docker suite running on host A. Host A also has a /data/clearml folder mounted to it and to it's docker containers (I've edited the docker-compose to add this mount point) Connect to host A, use StorageManager on the /data/clearml folder for some early troubles (e.g. long .list call) Use the same connection to run a task with execute_remotely and download_folder and see it crash š
š let me try to reproduce with it. can you write the change you edited in the docker-compose ?
Hi UnevenDolphin73 , what OS are you using? what is the clearml version?
UnevenDolphin73 I cant reproduce this issue on my side š can you give me some hints how to?
Any leads TimelyPenguin76 ? I've also tried setting up a minio s3 bucket, but I'm not sure if the remote agent has copied the credentials and host š¤
21s is just ridiculous, it's scanning the entire file system starting at /