I had several pipeline components getting it and uploading files to is concurrently.
Can Datsets handle that?
essentially, several running processes were performing:model_evals_dataset = Dataset.get( dataset_project=dataset_project, dataset_name=f"model_evals", ) model_evals_dataset.add_files(run_eval_path) model_evals_dataset.upload()
It seems to be doing ok on the app side:
I didn't realise Datasets had tasks associated with them but there is one and it seems to be doing ok.
I've attached it's log file which only mentions skipping one file (a warning)
Hi PanickyMoth78
I had several pipeline components getting it and uploading files to is concurrently.
Should not be a problem
I've attached it's log file which only mentions skipping one file (a warning)
So what exactly is the error you are getting?
from the screenshot it looks like a "gs://" access issue?
could that be it?
silly me. I deleted my gs credentials file :man-facepalming:
at least you did not change permission of your K8s etcd folder 😄