Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey Guys

Hey guys 😉 Couple months ago I was working on project based on ClearML and Pytorch Lightning and now I came back to it, upgraded ClearML and Lightning to the newest version and got stuck with a problem of no saving model (ckpt) into clearml - it was working back then. Other parameters and metrics are being saved correctly but clearml doesn't see this artifact (model) saved by Lightning. Have no idea whats going on. Do you have any idea why it doesn't work?

Upgraded packges:

  • pytorch lightning (1.8.0 -> 2.0.1)
  • clearml (1.7.2 -> 1.10.2)

I tried to reconstruct this example ( None ) but it is out-of-date.

  
  
Posted 11 months ago
Votes Newest

Answers 8


Good catch 🙂 my mistake. It was 1.7.2 (edited)

  
  
Posted 11 months ago

I've upgraded version from 1.7.2 to 1.10.2 and the problem has occured with the latest version. I have a feeling that it is related to major changes in Lightning with their 2+ version but its only my intuition. It would be nice if you could check it with your simple example code. Looking forward to the updates, thanks.

  
  
Posted 11 months ago

Hi @<1554638160548335616:profile|AverageSealion33> , are you sure the previous clearml version was 1.10.1? This version is only about a week old

  
  
Posted 11 months ago

Can you please check with the latest 1.10.2 SDK version if the checkpointing issue still happens. As for the example code which couldn't be reproduced, we're already working on it and should have a fix for it for the next minor SDK version

  
  
Posted 11 months ago

It may indeed be, thanks for letting us know, we’ll try to replicate it

  
  
Posted 11 months ago

Hey, yes, the reason for this issue seems to be our currently limited support for lightning 2.0. We will improve the support in the following releases. Right now one way to circumvent this issue, that I can recommend, is to use torch.save if possible, because we fully support automatic model capture on torch.save calls.

  
  
Posted 10 months ago

hi, any updates on this?

  
  
Posted 10 months ago

Hey, just for your information:
I replicated pytorch_lightning_example.py and it works fine (model/artifact saving) with old lightning interface (I mean "import pytorch_lightning as pl" as it is right now in the example) but the issue occurs when I try to use new lightning interface ("import lightning.pytorch as pl"). Hope it helps somehow and still looking forward to fix 🙂

  
  
Posted 11 months ago
574 Views
8 Answers
11 months ago
10 months ago
Tags