ok, for mayor version upgrade my suggestion is to backup the data somewhere and do a clean install after removing the pvc/pv

Do you have Ingresses enabled?

oh now I understand

with that in place k8s should be able to provisione pvc

I mean this blob is then saved on the fs

yaya confirmed

you need to investigate why it’s still in Pending state

but I think this behaviour will hange in future releases

ok so they are executed as expected

uh, using clearml-task params 😄

output_dest works:)

About last point: I would like to retrieve the pipeline 'output_dest' and use it as a parameter in adding steps, is that possible?

I have multiple agents not sharing /root/.trains

ok this is weird, in apiserver we should see call for deletion request. I need to consult with some people because I don’t think this is infra config related.

apiserver: additionalConfigs: services.conf: |should be
apiserver: additionalConfigs: apiserver.conf: |in this way he pod will mount a file called apiserver.conf instead of services.conf that is not the right filename for auth.

Did you use any override values.yaml file for this installation ?

This is pretty weird. If pv containing mongodb data is still the same data must be there. what storageclass provider are you using?

what kind of storageclass are you using on this one?

yes, exactly, agent creates and manages task pod lifecycle

there’s anything specific you need?

I didn’t get the point, if you need to change it, you can just override image name and tag by Helm

Sure, OddShrimp85 , until you need to specifically bind any pod to a node, nodeSelector is not needed. In fact, the new chart will leave to k8s the right to share the load on the worker nodes. About pvc you simply need to declare the Storageclass at k8s level so it can take care of creating the PVC too. How many workers do you have in your setup?

but I can be wrong, give me 30 mins while I recreate same local installation with samecchart so I can see if something is wrong

apiServerUrlReference: " "

ok next step is to exec into mongodb pod and check the various docscollections(tables) to see if data is still there

I suspect no for some reason that is related pvc management, at least for what I know

this is strange, I have a lot of clusters that went trough nodes issue but I never lost data

doubled copy paste

I’m not sure why in your case liveness probe is trying to access a non localhost ip. What is the version of the chart you are trying to install? helm list -A

but again, they are infrastructural decisions that need to be taken before talking software

With Helm we are not running in service-mode. If pod get evicted or killed we should investigate what is the reason behind that; there are any logs on kille dpod that can help us understand better the situation?