Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey There! I'M Encountering An Odd Issue - I'M Running My Agents As Python Processes On A Windows Pc Endpoints. I Recently Had A Bug That Forced Me To Delete All Cache And All (Non-Core) Venv-Builds. My Firstly Booted Agent Uses The ''First'' Venv-Build

hey there! I'm encountering an odd issue - I'm running my agents as python processes on a windows PC endpoints.

I recently had a bug that forced me to delete all cache and all (non-core) venv-builds. My firstly booted agent uses the ''first'' venv-build - the one without a .<num> appended to its dir name - And for some reason it already holds an empty .git directory representing the local clone of the git repo im working from. I can't delete this dir, because it says its being used by a different process in the system.

Then, every time I try to executre tasks on the queue assigned to this worker - It tries to clone the repo (since it's details for the worker it doesn't have the repo file), and fails - sinve that dir already exists and cant be deleted.
I'd love for some insights if someone might find some - As I'm currently blocked on this.
Thank you!

  
  
Posted 2 months ago
Votes Newest

Answers 11


but again - The system denies my deletion requiest since it deems the venv-builds dir as in use

  
  
Posted 2 months ago

Does this provide any more context? @<1523701087100473344:profile|SuccessfulKoala55>

  
  
Posted 2 months ago

Hi @<1523701087100473344:profile|SuccessfulKoala55> - Each worker uses its own venv-builds folder

  
  
Posted 2 months ago

From your description it seems what fails is setting the vcs cache? i.e. copying the cloned repository?

  
  
Posted 2 months ago

is would definetly seem that way - Although when I look at the error logs, the failure is actually in creating the venv-build folder - (under the task_repository subfolder) -

Repository cloning failed: [WinError 183] Cannot create a file when that file already exists: 'C:\\Users\\clearml-admin\\.clearml\\venvs-builds\\3.1\\task_repository\\<repo>.git'

  
  
Posted 2 months ago

huh. It is weird. is there any way to force deletion of it? it seems its still being held be some task and the server has been restarted several times since

  
  
Posted 2 months ago

The system denies my deletion requiest since it deems the venv-builds dir as in use

Sorry, yes you have to take down the agent when you delete the cache 🙂

  
  
Posted 2 months ago

@<1710827340621156352:profile|HungryFrog27> the venv-build folder is supposed to be deleted after each task is done. How did you end up with leftovers? Could it be windows was failing to delete it for some reason? That actually connects with you initial issue no?

  
  
Posted 2 months ago

I can

  
  
Posted 2 months ago

is it a shared network mount ? could you just delete the entire ~/.clearml on the host machine ?

  
  
Posted 2 months ago

Hi @<1710827340621156352:profile|HungryFrog27> , are all of your agents using the same venv-builds folder?

  
  
Posted 2 months ago
236 Views
11 Answers
2 months ago
2 months ago
Tags