What is the efficient way of doing this?

We are using matplotlib in some tasks

I copied the Google Storage part clearml.conf and creds.json in the agent to my local and now it gets stuck here if I run:

Exception has occurred: ValueError Upload failed
Code:
# Upload separated files for separated_file in Path(dataset_folder).rglob(f'*{batch_id}*'): remote_file_path = f"gs://<bucket>/{hatchery_id}/{batch_id}/{single_egg_object_id}/{folder_type}/{str(PurePath(separated_file)).split('/')[-1]}" StorageManager.upload_file(local_file=separated_file, remote_url=remote_file_path)

For separated_file_posix_path

Actually the values of separated_file_posix_path and separated_file are same

Yes I was passing that. Now I am passing it the same way you have mentioned, but my code still gets stuck as in above screenshot. My guess is that it can't resolve credentials. It does not give me any pop up to login also

Just added. What about output_uri?

Yes, my bad. It was VPN issue and I fixed it. Now I have one last error:
[Errno 2] No such file or directory: '<filename.extension>'

from clearml import Task, Dataset, StorageManager dataset_rgb = Dataset.get(dataset_id=args['dataset_id_rgb']) folder_rgb = dataset_rgb.get_local_copy() files_rgb = dataset_rgb.list_files() for separated_file in files_rgb: separated_file_posix_path = PosixPath(separated_file) remote_file_path = f"gs://<bucket-name>/<folder-name>" StorageManager.upload_file(local_file=separated_file_posix_path, remote_url=remote_file_path)

remote_url = 'gs://<bucket>/6141af10b81a76709db962fd/6141b0ee5f627d0020819489/6147c3db1f0d7f48ddf8e952/camera/6141b0ee5f627d0020819489_14.png' local_file = '6141b0ee5f627d0020819489_14.png'

Yes it is. They come always for us. Even if there is no matplotlib in requirements

We are uploading it to Google Bucket because this is the last step of processing after image acquisition and before ML modelling. We need a more permanent place to store data

I need files in different folders. So I cannot upload whole folder

The screenshot is from the clearml server. Sometimes I run the pipeline on server and monitor it. If a task has failed, I want to copy these parameters and debug the task locally. But clearml does not allow me to copy them. So I have to download this as a JSON and search for the params there