Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone! I'Ve Had A Problem. But When I Was Describing It Here It Was Solved. Maybe It Will Help Someone. I Use Pytorch And Training Accidentally Freezes After Weights Uploading By Trains. Don'T Know Exactly What'S Wrong, But It Was Somehow Connected

Hi everyone!
I've had a problem. But when I was describing it here it was solved.

Maybe it will help someone. I use pytorch and training accidentally freezes after weights uploading by trains. Don't know exactly what's wrong, but it was somehow connected with multithreding in pytorch dataloaders and cv2. These magic lines have solved the problem
if  dataloader.num_workers  >  0:
cv2.setNumThreads(0)

  
  
Posted 4 years ago
pic
More
PungentJellyfish55
0 × 1
Report
Votes Newest

Answers 7

PungentLouse55 could you test with 0.15.2rc0 see if there is any difference ?

  
  
Posted 4 years ago
pic
More
AgitatedDove14
0 × 1
Report

0.15.1

  
  
Posted 4 years ago
pic
More
PungentJellyfish55
0 × 1
Report

It doesn't not seem to be related to the upload. The upload itself finished... What's your Trains version?

  
  
Posted 4 years ago
pic
More
AgitatedDove14
0 × 1
Report

Unfortunately I still have the same issue 😢

Here is the stack on manually interupting (wights were uploaded on 14:16 and I interrupted on 14:30)

  
  
Posted 4 years ago
pic
More
PungentJellyfish55
0 × 1
Report

PungentLouse55 hmmm
Do you have an idea on how we could quickly reproduce it?

  
  
Posted 4 years ago
pic
More
AgitatedDove14
0 × 1
Report

I don't know. It looked like an ordinary weights uploading. Here's the screenshot

  
  
Posted 4 years ago
pic
More
PungentJellyfish55
0 × 1
Report

Hi PungentLouse55 ,
I think can see how these magic lines solved it, and I think you are onto something.
Any chance what happened is multiple workers were trying to simultaneously save/load the same Model ?

  
  
Posted 4 years ago
pic
More
AgitatedDove14
0 × 1
Report
Write your answer
998 Views
7 Answers
4 years ago
one year ago
Tags
pytorch