I don't know, I only see the error

ERROR: Could not find a version that satisfies the requirement pandas==1.2.4 (from -r /tmp/cached-reqs1i9hihyv.txt (line 14)) (from versions: 0.1, 0.2b0, 0.2b1, 0.2, 0.3.0b0, 0.3.0b2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0rc1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0rc1, 0.8.0rc2, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0rc1, 0.19.0, 0.19...

For example, the attached package is being skipped automatically but it is needed while running the actual script

actually I should rephrase my question

yea the api server configuration also went away

no that's for sure not

Yes. Sorry, I have used the wrong backup for some reason

Do you mind if I send it in a private chat?

I will have to remove some parts because it contains some sensitive data

ohh sec

I have another problem at the moment (some packages that are not inside the cache seem to have be missing and then everything fails)

it woudl be the same with a docker container and -v then 🙂 np

I tried to add it to post_packages and run it locally, for some reason it didn't install googlecloud-storage. However it is possible that I have an old clearml pip package installed

check if you have any more of those recovery reports in the mongo log, it should report progress

I think I have sent you all the existing logs

I see that there's a system_site_packages: false,

that would be a great solution

going to try it now

That's a cool idea. Then you pass the tolerations definition through a different pod template?

ERROR: No matching distribution found for pandas==1.2.4 (from -r /tmp/cached-reqs1i9hihyv.txt (line 14))

azure

Ok

yea

you can actually inject a known_hosts file to your docker container/k8s pod through a volume

on my own machine

nothing for now

I will investigate a bit more and then check if I can recover

so if the node went down and then some other node came up, the data is lost

AgitatedDove14 so basically I am using my own docker image with all of our internal dependencies already installed, including python packages. Some of these packages (e.g green-common ) should be loaded automatically from the dockerimage.

I thought that maybe the virtualenv should https://stackoverflow.com/questions/12079607/make-virtualenv-inherit-specific-packages-from-your-global-site-packages --system-site-packages https://stackoverflow.com/questions/12079607/make-virtualenv-...

I hope so

but the PV seems to be just a path to the labeled node