It seems to be doing ok on the app side:
I didn't realise Datasets had tasks associated with them but there is one and it seems to be doing ok.
I've attached it's log file which only mentions skipping one file (a warning)

sorry..

I had several pipeline components getting it and uploading files to is concurrently.
Can Datsets handle that?

No worries 🙂

from the screenshot it looks like a "gs://" access issue?
could that be it?

at least you did not change permission of your K8s etcd folder 😄

Hi PanickyMoth78

I had several pipeline components getting it and uploading files to is concurrently.

Should not be a problem

I've attached it's log file which only mentions skipping one file (a warning)

So what exactly is the error you are getting?

essentially, several running processes were performing:
model_evals_dataset = Dataset.get( dataset_project=dataset_project, dataset_name=f"model_evals", ) model_evals_dataset.add_files(run_eval_path) model_evals_dataset.upload()

silly me. I deleted my gs credentials file :man-facepalming: