Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello Everyone! We'Ve Been Using Clearml For A Couple Weeks Now With Everything Working Out Fine, But In Recent Days We'Ve Run Into An Issue With Parameter Optimizer Tasks - Without Any Apparent Change On Our Side, All Child Tasks Of The Optimizer Are Abo

Hello everyone!
We've been using ClearML for a couple weeks now with everything working out fine, but in recent days we've run into an issue with parameter optimizer tasks - Without any apparent change on our side, all child tasks of the optimizer are aborting during the environment setting (while attempting to download the files), and without starting to log the task using clearml.
Any ideas on what might lead to this?
Thanks!
image

  
  
Posted 6 months ago
pic
More
HungryFrog27
0 × 1
Report
Votes Newest

Answers 6

@<1523701087100473344:profile|SuccessfulKoala55> Thanks, I'll check around that!

  
  
Posted 5 months ago
pic
More
HungryFrog27
0 × 1
Report

Hey!
I see in my agent debug logs that it's constantly dropping the connection with the ClearML Server. I also see my tasks being aborted as User aborted (3) - Just at the point where the (post requirements) venv is added into the local venv cache. Could there be any connection? And if not, does anyone have any clue as to where to continue my debugging?
image
image

  
  
Posted 5 months ago
pic
More
HungryFrog27
0 × 1
Report

@<1710827340621156352:profile|HungryFrog27> (3) in this case means the agent detected that the task's state was changed from running to something else (the agent checks that 3 times to make sure, waiting 3 seconds between each check)

  
  
Posted 5 months ago
pic
More
SuccessfulKoala55
0 × 1
Report

Hey @<1523701087100473344:profile|SuccessfulKoala55> , 1.8.1

  
  
Posted 5 months ago
pic
More
HungryFrog27
0 × 1
Report

Hi @<1710827340621156352:profile|HungryFrog27> , this should have relation to the setup of the venv. What agent version are you using?

  
  
Posted 5 months ago
pic
More
SuccessfulKoala55
0 × 1
Report

Hi @<1710827340621156352:profile|HungryFrog27> , I'd suggest running the agent with --debug flag for more information. Can you provide a full log of both the HPO task and one of the children?

  
  
Posted 6 months ago
pic
More
CostlyOstrich36
0
Report
Write your answer
482 Views
6 Answers
6 months ago
5 months ago
Tags
clearml