Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Good Day! I Ran Into A Problem When When Running Two Or More Identical Nodes In A Pipeline (Multi_Instance_Support=True ), Only One Of Them Uses An Already Created Venv From Cache For This Task. And The Other Node Starts To Re-Create The Same Virtual Envi

Good day! I ran into a problem when when running two or more identical nodes in a pipeline (multi_instance_support=True ), only one of them uses an already created venv from cache for this task. And the other node starts to re-create the same virtual environment.

The error is this:

WARNING:root:Could not lock cache folder /home/user/.clearml/venvs-cache: [Errno 11] Resource temporarily unavailable 

Is it possible to make them sequentially access the already created virtual environment. Go something like that?

ubuntu 20.04
Latest version of clearml

  
  
Posted 6 months ago
Votes Newest

Answers 6


I'm sorry, I must have misunderstood you. They are both trying to use venv from the cache. Let's say I need to run 10 identical nodes at the same time. The necessary ven is already in the cache. They are independent of each other, but have the same set of libraries. This venv from the cache is suitable for all of them. But only 1 node will copy it. Everyone else will start doing exactly the same venv, as if it is not in the cache.

  
  
Posted 6 months ago

Understood, thank you!

  
  
Posted 6 months ago

Hi @<1598487094601191424:profile|MysteriousCow84>

only one of them uses an already created venv from cache for this task. And the other node starts to re-create the same virtual environment.

Just be clear, the second one is running, but it does not use the same venv as the other one (that is running in parallel), is that correct?

  
  
Posted 6 months ago

This is by design, they cannot use the exact same venv because if the code starts creating files/change them it happens inside the venv and might cause them to crash.
That said if you are running with venv cache, the first one will create the venv and the second one will create a copy from the cache.

  
  
Posted 6 months ago

But only 1 node will copy it.

they can only copy it after the first is finished, and they are not aware it is trying to set the exact venv, hence the race

  
  
Posted 6 months ago

Yes, both nodes are running at the same time. The 1st uses the desired venv. The 2nd creates exactly the same venv anew

  
  
Posted 6 months ago
436 Views
6 Answers
6 months ago
6 months ago
Tags