Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Clearml Doesn'T Pick Up Model Checkpoints Automatically. Any Idea What Might Be Wrong? (Code Attached In The Thread). Thanks

ClearML doesn't pick up model checkpoints automatically. Any idea what might be wrong? (code attached in the thread). Thanks

  
  
Posted 4 months ago
Votes Newest

Answers 11


Which version? is this reproducible in this example?
None
(can you try with the latest clearml version 1.13.2?)

  
  
Posted 3 months ago

Oh, I misunderstood then docs/examples, sorry. I'm using pytorch-ignite.

Thanks for the tip!

  
  
Posted 3 months ago

Hi @<1631102016807768064:profile|ZanySealion18>

ClearML doesn't pick up model checkpoints automatically.

What's the framework you are using?
BTW:

Task.add_requirements("requirements.txt")

if you want to specify Just your requirements.txt, do not use add_requirements use:

Task.force_requirements_env_freeze(requirements_file="requirements.txt")

(add requirements with a filename does the same thing, but this is more readable)

  
  
Posted 3 months ago

@<1523701087100473344:profile|SuccessfulKoala55> Kind reminder again, thanks and sorry!

  
  
Posted 3 months ago

@<1523701087100473344:profile|SuccessfulKoala55> kind reminder not to miss this when you catch time, thanks!

  
  
Posted 3 months ago

No worries, sorry for pinging, was just making sure you (or anyone else who might help) doesn't miss it 🙂
I use Task.add_requirements("requirements.txt") right before the Task.init.
In main, I parse arguments command-line, add_requirements, initialize Task and call execute_remotely. After that it's all pretty much the usual workflow. Initialize the model, setup dataloaders, optimizer and run the training. I'm using pytorch-ignite and have model checkpoint made on validation evaluator COMPLETED event.

  
  
Posted 3 months ago

Sorry for the delay 🙏 - how do you import your packages and where do you initialize ClearML relative to the rest of the code?

  
  
Posted 3 months ago

Kind ping on this thread, thanks! 🙂

  
  
Posted 3 months ago

clearml-1.13.1

Task.add_requirements("requirements.txt")
task = Task.init(project_name="My project", task_name="My task")
task.execute_remotely(queue_name="default")
...
  
  
Posted 4 months ago

Hi @<1631102016807768064:profile|ZanySealion18> , can you provide more info on what framework you're using, which ClearML SDK version and how you're initializing the ClearML task?

  
  
Posted 4 months ago

    model_checkpoint = ModelCheckpoint(
        "checkpoint",
        n_saved=2,
        filename_prefix="best",
        score_function=score_function,
        score_name="accuracy",
        global_step_transform=global_step_from_engine(trainer),
    )

    # Save the model after every epoch of val_evaluator is completed
    val_evaluator.add_event_handler(
        Events.COMPLETED, model_checkpoint, {"model": model}
    )
  
  
Posted 4 months ago
353 Views
11 Answers
4 months ago
3 months ago
Tags