I'm sorry, I must have misunderstood you. They are both trying to use venv from the cache. Let's say I need to run 10 identical nodes at the same time. The necessary ven is already in the cache. They are independent of each other, but have the same set of libraries. This venv from the cache is suitable for all of them. But only 1 node will copy it. Everyone else will start doing exactly the same venv, as if it is not in the cache.

But only 1 node will copy it.

they can only copy it after the first is finished, and they are not aware it is trying to set the exact venv, hence the race

Yes, both nodes are running at the same time. The 1st uses the desired venv. The 2nd creates exactly the same venv anew

Understood, thank you!

Hi @<1598487094601191424:profile|MysteriousCow84>

only one of them uses an already created venv from cache for this task. And the other node starts to re-create the same virtual environment.

Just be clear, the second one is running, but it does not use the same venv as the other one (that is running in parallel), is that correct?

This is by design, they cannot use the exact same venv because if the code starts creating files/change them it happens inside the venv and might cause them to crash.
That said if you are running with venv cache, the first one will create the venv and the second one will create a copy from the cache.