I see that there's a system_site_packages: false,

I don't know, I only see the error

I hope I didn't miss anything

Should I make a new issue or just reply on the one I mentioned above?

that would be a great solution

ERROR: Could not find a version that satisfies the requirement pandas==1.2.4 (from -r /tmp/cached-reqs1i9hihyv.txt (line 14)) (from versions: 0.1, 0.2b0, 0.2b1, 0.2, 0.3.0b0, 0.3.0b2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0rc1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0rc1, 0.8.0rc2, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0rc1, 0.19.0, 0.19...

ERROR: No matching distribution found for pandas==1.2.4 (from -r /tmp/cached-reqs1i9hihyv.txt (line 14))

check if you have any more of those recovery reports in the mongo log, it should report progress

I think I have sent you all the existing logs

Yea definitely

For example, the attached package is being skipped automatically but it is needed while running the actual script

actually I should rephrase my question

OK I added:

- name: TRAINS_CONFIG_FILE value: "/secrets/clearml.conf"
to my pod template, and this file exists and mounted as expected but it still fails. I will see if I missed something

but the PV seems to be just a path to the labeled node

nothing for now

azure

You can either use the StrictHostKeyChecking=no or generate a known_hosts file. I don't know about other options

Ok

it woudl be the same with a docker container and -v then 🙂 np

Seems like this is not the best solution:

why not?

Do you mind if I send it in a private chat?

I will investigate a bit more and then check if I can recover

thank you for your time and support, I appreciate it!

Now I suspect what happened is it stayed on another node, and your k8s never took care of that

that's an interesting theory

Yes. Sorry, I have used the wrong backup for some reason

well, it's only when adding a - name to the template

AgitatedDove14 so basically I am using my own docker image with all of our internal dependencies already installed, including python packages. Some of these packages (e.g green-common ) should be loaded automatically from the dockerimage.

I thought that maybe the virtualenv should https://stackoverflow.com/questions/12079607/make-virtualenv-inherit-specific-packages-from-your-global-site-packages --system-site-packages https://stackoverflow.com/questions/12079607/make-virtualenv-...

Could it be it was never allocated to begin with ?

what do you mean?

You need to mount it to 

~/clearml.conf

 (i.e. /root/clearml.conf)

I have overridden that with the env var

so if the node went down and then some other node came up, the data is lost

anyways, it works. I will update the issue