Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All. I Am Using The Recently Added Trainslogger In Pytorch-Lightning And Experiencing Incoherent Behavior With Model Checkpoint Upload. I Made An Issue On Pytorch-Lightning Github

Hi all. I am using the recently added TrainsLogger in Pytorch-Lightning and experiencing incoherent behavior with model checkpoint upload. I made an issue on Pytorch-Lightning Github https://github.com/PyTorchLightning/pytorch-lightning/issues/1466#issuecomment-612633043 but I wonder if there's any ideas how to fix this from Trains side.
In short, Lightning saves model as a .ckpt.part file, then Trains starts asynchronous upload to output_uri , but at the same time Lightning renames the file to .ckpt

  
  
Posted 3 years ago
Votes Newest

Answers 8


Great, thanks 🙂

  
  
Posted 3 years ago

MelancholyBeetle72 thanks! I'll see if we could release an RC with a fix soon, for you to test :)

  
  
Posted 3 years ago

Hi MelancholyBeetle72 , that's a very interesting case. I can totally understand how storing a model and then immediately renaming it breaks the upload. A few questions, is there a way for pytorch lightning not to rename the model? Also I wonder if this scenario happens a lot (storing model and changing it) . I think the best solution is for Trains to create a copy of the file and upload it in the background. That said the name will still end with .part What do you think?

  
  
Posted 3 years ago

is there a way for pytorch lightning not to rename the model?

I think no, since pytorch-lightning ensures atomarity of save operation by this logic.

I think the best solution is for Trains to create a copy of the file and upload it in the background

Yes, sounds good.
Yeah, I'll open an issue on Trains Github

  
  
Posted 3 years ago

MelancholyBeetle72 there is an RC with a fix, check the GitHub issue for details :)

  
  
Posted 3 years ago

MelancholyBeetle72 it will be great if you could also open an issue on Trains and reference the pytorch lightning issue, could you please?

  
  
Posted 3 years ago

Wow, astonishingly fast 🙂
Ok, I'll check my case and leave feedback

  
  
Posted 3 years ago
783 Views
8 Answers
3 years ago
one year ago
Tags