Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
We'Re Trying To Upgrade Our Clearml On K8S But We'Re Getting This Error -

We're trying to upgrade our ClearML on K8s but we're getting this error -
Error: UPGRADE FAILED: cannot patch "clearml-fileserver-data" with kind PersistentVolumeClaim: PersistentVolumeClaim "clearml-fileserver-data" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claimsWhat should we do 🤔
For now it was a near-empty install so we also added a deployment step with state: absent (with use ansible), and that obviously removed anything we had there previously.

  
  
Posted one year ago
Votes Newest

Answers 23


about minor releases they are not breaking so it should be linear

  
  
Posted one year ago

Of course. We'd like to use S3 backends anyway, I couldn't spot exactly where to configure this in the chart (so it's defined in the individual agent's configuration)

  
  
Posted one year ago

apiserver additionalConfigs

  
  
Posted one year ago

https://artifacthub.io/packages/helm/allegroai/clearml and click on install button so you can see details about repo

  
  
Posted one year ago

Full log:
command: /usr/sbin/helm --version=4.1.2 upgrade -i --reset-values --wait -f=/tmp/tmp77d9ecye.yml clearml clearml/clearml msg: |- Failure when executing Helm command. Exited 1. stdout: stderr: W0728 09:23:47.076465 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget W0728 09:23:47.126364 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget W0728 09:23:47.188124 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget Error: UPGRADE FAILED: cannot patch "clearml-fileserver-data" with kind PersistentVolumeClaim: PersistentVolumeClaim "clearml-fileserver-data" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims core.PersistentVolumeClaimSpec{ ... // 2 identical fields Resources: {Requests: {s"storage": {i: {...}, s: "50Gi", Format: "BinarySI"}}}, VolumeName: "", - StorageClassName: nil, + StorageClassName: &"standard", VolumeMode: &"Filesystem", DataSource: nil, DataSourceRef: nil, } stderr: |- W0728 09:23:47.076465 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget W0728 09:23:47.126364 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget W0728 09:23:47.188124 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget Error: UPGRADE FAILED: cannot patch "clearml-fileserver-data" with kind PersistentVolumeClaim: PersistentVolumeClaim "clearml-fileserver-data" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims core.PersistentVolumeClaimSpec{ ... // 2 identical fields Resources: {Requests: {s"storage": {i: {...}, s: "50Gi", Format: "BinarySI"}}}, VolumeName: "", - StorageClassName: nil, + StorageClassName: &"standard", VolumeMode: &"Filesystem", DataSource: nil, DataSourceRef: nil, } stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>

  
  
Posted one year ago

Hi UnevenDolphin73 , maybe JuicyFox94 or SuccessfulKoala55 can assist

  
  
Posted one year ago

For now this is okay - no data lost, really - but I'd like to make sure we're not missing any steps in the next upgrade

  
  
Posted one year ago

Could you provide a more complete set of instructions, for the less inclined?
How would I backup the data in future times etc?

  
  
Posted one year ago

This is specific K8s infra management, usually I use Velero for backup

  
  
Posted one year ago

if yopu instruct apiserver to use s3 fileserver will not basically used anymore (I need SuccessfulKoala55 confirmation to be 100% sure, Im more infra guy :D )

  
  
Posted one year ago

ok, for mayor version upgrade my suggestion is to backup the data somewhere and do a clean install after removing the pvc/pv

  
  
Posted one year ago

Hm, I'm not sure I follow 🤔 How does the API server config relate to the file server?

  
  
Posted one year ago

check values.yaml

  
  
Posted one year ago

but again, they are infrastructural decisions that need to be taken before talking software

  
  
Posted one year ago

At any case, if we were upgrading from e.g. 4.0.0 to 4.1.2, this shouldn't have happened?

  
  
Posted one year ago

there are workarounds tbh but they are tricks that require a lot of k8s espertise and they are risky

  
  
Posted one year ago

moreover I usually prefer to have S3 backends if in AWS oir MinIO

  
  
Posted one year ago

Okay, I'll test it out by trying to downgrade to 4.0.0 and then upgrade to 4.1.2
Just to make sure, the chart_ref is allegroai/clearml right? (for some reason we had clearml/clearml and it seems like it previously worked?)

  
  
Posted one year ago

What Chart version are you trying to upgrade from?

  
  
Posted one year ago

Removing the PVC is just setting the state to absent AFAIK

  
  
Posted one year ago

you can create a specific config like one in https://clear.ml/docs/latest/docs/integrations/storage/

  
  
Posted one year ago

3.3.0 😅

  
  
Posted one year ago

not sure if this is ok with your infra

  
  
Posted one year ago
605 Views
23 Answers
one year ago
one year ago
Tags