Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
We'Re Trying To Upgrade Our Clearml On K8S But We'Re Getting This Error -

We're trying to upgrade our ClearML on K8s but we're getting this error -
Error: UPGRADE FAILED: cannot patch "clearml-fileserver-data" with kind PersistentVolumeClaim: PersistentVolumeClaim "clearml-fileserver-data" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claimsWhat should we do 🤔
For now it was a near-empty install so we also added a deployment step with state: absent (with use ansible), and that obviously removed anything we had there previously.

  
  
Posted 2 years ago
Votes Newest

Answers 23


https://artifacthub.io/packages/helm/allegroai/clearml and click on install button so you can see details about repo

  
  
Posted 2 years ago

ok, for mayor version upgrade my suggestion is to backup the data somewhere and do a clean install after removing the pvc/pv

  
  
Posted 2 years ago

there are workarounds tbh but they are tricks that require a lot of k8s espertise and they are risky

  
  
Posted 2 years ago

What Chart version are you trying to upgrade from?

  
  
Posted 2 years ago

This is specific K8s infra management, usually I use Velero for backup

  
  
Posted 2 years ago

moreover I usually prefer to have S3 backends if in AWS oir MinIO

  
  
Posted 2 years ago

Removing the PVC is just setting the state to absent AFAIK

  
  
Posted 2 years ago

not sure if this is ok with your infra

  
  
Posted 2 years ago

For now this is okay - no data lost, really - but I'd like to make sure we're not missing any steps in the next upgrade

  
  
Posted 2 years ago

if yopu instruct apiserver to use s3 fileserver will not basically used anymore (I need SuccessfulKoala55 confirmation to be 100% sure, Im more infra guy :D )

  
  
Posted 2 years ago

At any case, if we were upgrading from e.g. 4.0.0 to 4.1.2, this shouldn't have happened?

  
  
Posted 2 years ago

about minor releases they are not breaking so it should be linear

  
  
Posted 2 years ago

you can create a specific config like one in https://clear.ml/docs/latest/docs/integrations/storage/

  
  
Posted 2 years ago

Okay, I'll test it out by trying to downgrade to 4.0.0 and then upgrade to 4.1.2
Just to make sure, the chart_ref is allegroai/clearml right? (for some reason we had clearml/clearml and it seems like it previously worked?)

  
  
Posted 2 years ago

Hm, I'm not sure I follow 🤔 How does the API server config relate to the file server?

  
  
Posted 2 years ago

Could you provide a more complete set of instructions, for the less inclined?
How would I backup the data in future times etc?

  
  
Posted 2 years ago

but again, they are infrastructural decisions that need to be taken before talking software

  
  
Posted 2 years ago

apiserver additionalConfigs

  
  
Posted 2 years ago

check values.yaml

  
  
Posted 2 years ago

Of course. We'd like to use S3 backends anyway, I couldn't spot exactly where to configure this in the chart (so it's defined in the individual agent's configuration)

  
  
Posted 2 years ago

Hi UnevenDolphin73 , maybe JuicyFox94 or SuccessfulKoala55 can assist

  
  
Posted 2 years ago

3.3.0 😅

  
  
Posted 2 years ago

Full log:
command: /usr/sbin/helm --version=4.1.2 upgrade -i --reset-values --wait -f=/tmp/tmp77d9ecye.yml clearml clearml/clearml msg: |- Failure when executing Helm command. Exited 1. stdout: stderr: W0728 09:23:47.076465 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget W0728 09:23:47.126364 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget W0728 09:23:47.188124 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget Error: UPGRADE FAILED: cannot patch "clearml-fileserver-data" with kind PersistentVolumeClaim: PersistentVolumeClaim "clearml-fileserver-data" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims core.PersistentVolumeClaimSpec{ ... // 2 identical fields Resources: {Requests: {s"storage": {i: {...}, s: "50Gi", Format: "BinarySI"}}}, VolumeName: "", - StorageClassName: nil, + StorageClassName: &"standard", VolumeMode: &"Filesystem", DataSource: nil, DataSourceRef: nil, } stderr: |- W0728 09:23:47.076465 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget W0728 09:23:47.126364 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget W0728 09:23:47.188124 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget Error: UPGRADE FAILED: cannot patch "clearml-fileserver-data" with kind PersistentVolumeClaim: PersistentVolumeClaim "clearml-fileserver-data" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims core.PersistentVolumeClaimSpec{ ... // 2 identical fields Resources: {Requests: {s"storage": {i: {...}, s: "50Gi", Format: "BinarySI"}}}, VolumeName: "", - StorageClassName: nil, + StorageClassName: &"standard", VolumeMode: &"Filesystem", DataSource: nil, DataSourceRef: nil, } stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>

  
  
Posted 2 years ago
992 Views
23 Answers
2 years ago
one year ago
Tags