Hi PanickyMoth78
I had several pipeline components getting it and uploading files to is concurrently.
Should not be a problem
I've attached it's log file which only mentions skipping one file (a warning)
So what exactly is the error you are getting?
essentially, several running processes were performing:model_evals_dataset = Dataset.get( dataset_project=dataset_project, dataset_name=f"model_evals", ) model_evals_dataset.add_files(run_eval_path) model_evals_dataset.upload()
I had several pipeline components getting it and uploading files to is concurrently.
Can Datsets handle that?
It seems to be doing ok on the app side:
I didn't realise Datasets had tasks associated with them but there is one and it seems to be doing ok.
I've attached it's log file which only mentions skipping one file (a warning)
at least you did not change permission of your K8s etcd folder 😄
from the screenshot it looks like a "gs://" access issue?
could that be it?
silly me. I deleted my gs credentials file :man-facepalming: