from clearml import Task, Dataset, StorageManager dataset_rgb = Dataset.get(dataset_id=args['dataset_id_rgb']) folder_rgb = dataset_rgb.get_local_copy() files_rgb = dataset_rgb.list_files() for separated_file in files_rgb: separated_file_posix_path = PosixPath(separated_file) remote_file_path = f"gs://<bucket-name>/<folder-name>" StorageManager.upload_file(local_file=separated_file_posix_path, remote_url=remote_file_path)

Yes I was passing that. Now I am passing it the same way you have mentioned, but my code still gets stuck as in above screenshot. My guess is that it can't resolve credentials. It does not give me any pop up to login also

We are uploading it to Google Bucket because this is the last step of processing after image acquisition and before ML modelling. We need a more permanent place to store data

Exception has occurred: ValueError Upload failed
Code:
# Upload separated files for separated_file in Path(dataset_folder).rglob(f'*{batch_id}*'): remote_file_path = f"gs://<bucket>/{hatchery_id}/{batch_id}/{single_egg_object_id}/{folder_type}/{str(PurePath(separated_file)).split('/')[-1]}" StorageManager.upload_file(local_file=separated_file, remote_url=remote_file_path)

Actually the values of separated_file_posix_path and separated_file are same

For separated_file_posix_path

Just added. What about output_uri?

What is the efficient way of doing this?

Yes, my bad. It was VPN issue and I fixed it. Now I have one last error:
[Errno 2] No such file or directory: '<filename.extension>'

I copied the Google Storage part clearml.conf and creds.json in the agent to my local and now it gets stuck here if I run:

The screenshot is from the clearml server. Sometimes I run the pipeline on server and monitor it. If a task has failed, I want to copy these parameters and debug the task locally. But clearml does not allow me to copy them. So I have to download this as a JSON and search for the params there

We are using matplotlib in some tasks

Yes it is. They come always for us. Even if there is no matplotlib in requirements

remote_url = 'gs://<bucket>/6141af10b81a76709db962fd/6141b0ee5f627d0020819489/6147c3db1f0d7f48ddf8e952/camera/6141b0ee5f627d0020819489_14.png' local_file = '6141b0ee5f627d0020819489_14.png'

I need files in different folders. So I cannot upload whole folder