nothing for now

so if the node went down and then some other node came up, the data is lost

I will investigate a bit more and then check if I can recover

no that's for sure not

I wonder if it's completely lost

that would be a great solution

you can actually inject a known_hosts file to your docker container/k8s pod through a volume

Seems like this is not the best solution:

why not?

so if you are using k8s, you can generate a configMap with the relevant info and mount it

You can either use the StrictHostKeyChecking=no or generate a known_hosts file. I don't know about other options

oh yea, that is true regarding mitm attacks

For example, the attached package is being skipped automatically but it is needed while running the actual script

thank you for your time and support, I appreciate it!

Now I suspect what happened is it stayed on another node, and your k8s never took care of that

that's an interesting theory

ERROR: Could not find a version that satisfies the requirement pandas==1.2.4 (from -r /tmp/cached-reqs1i9hihyv.txt (line 14)) (from versions: 0.1, 0.2b0, 0.2b1, 0.2, 0.3.0b0, 0.3.0b2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0rc1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0rc1, 0.8.0rc2, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0rc1, 0.19.0, 0.19...

Ok

I don't know, I only see the error

I will have to remove some parts because it contains some sensitive data

ERROR: No matching distribution found for pandas==1.2.4 (from -r /tmp/cached-reqs1i9hihyv.txt (line 14))

Do you mind if I send it in a private chat?

yea

I hope I didn't miss anything

I'll do that

Interesting point

Yea sure

I agree

Should I make a new issue or just reply on the one I mentioned above?

anyways, it works. I will update the issue

OK I added:

- name: TRAINS_CONFIG_FILE value: "/secrets/clearml.conf"
to my pod template, and this file exists and mounted as expected but it still fails. I will see if I missed something

seems like I miss configured something of course. /secrets is mounted to the agent but not to the pod template 🙂