I copied the Google Storage part clearml.conf and creds.json in the agent to my local and now it gets stuck here if I run:

from clearml import Task, Dataset, StorageManager dataset_rgb = Dataset.get(dataset_id=args['dataset_id_rgb']) folder_rgb = dataset_rgb.get_local_copy() files_rgb = dataset_rgb.list_files() for separated_file in files_rgb: separated_file_posix_path = PosixPath(separated_file) remote_file_path = f"gs://<bucket-name>/<folder-name>" StorageManager.upload_file(local_file=separated_file_posix_path, remote_url=remote_file_path)

For separated_file_posix_path

The screenshot is from the clearml server. Sometimes I run the pipeline on server and monitor it. If a task has failed, I want to copy these parameters and debug the task locally. But clearml does not allow me to copy them. So I have to download this as a JSON and search for the params there

Exception has occurred: ValueError Upload failed
Code:
# Upload separated files for separated_file in Path(dataset_folder).rglob(f'*{batch_id}*'): remote_file_path = f"gs://<bucket>/{hatchery_id}/{batch_id}/{single_egg_object_id}/{folder_type}/{str(PurePath(separated_file)).split('/')[-1]}" StorageManager.upload_file(local_file=separated_file, remote_url=remote_file_path)

remote_url = 'gs://<bucket>/6141af10b81a76709db962fd/6141b0ee5f627d0020819489/6147c3db1f0d7f48ddf8e952/camera/6141b0ee5f627d0020819489_14.png' local_file = '6141b0ee5f627d0020819489_14.png'

I need files in different folders. So I cannot upload whole folder

Just added. What about output_uri?

Yes, my bad. It was VPN issue and I fixed it. Now I have one last error:
[Errno 2] No such file or directory: '<filename.extension>'

What is the efficient way of doing this?

We are using matplotlib in some tasks

Yes it is. They come always for us. Even if there is no matplotlib in requirements

Yes I was passing that. Now I am passing it the same way you have mentioned, but my code still gets stuck as in above screenshot. My guess is that it can't resolve credentials. It does not give me any pop up to login also

Actually the values of separated_file_posix_path and separated_file are same

We are uploading it to Google Bucket because this is the last step of processing after image acquisition and before ML modelling. We need a more permanent place to store data