Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Clearml Doesn'T Pick Up Model Checkpoints Automatically. Any Idea What Might Be Wrong? (Code Attached In The Thread). Thanks

ClearML doesn't pick up model checkpoints automatically. Any idea what might be wrong? (code attached in the thread). Thanks

  
  
Posted 11 months ago
Votes Newest

Answers 11


Which version? is this reproducible in this example?
None
(can you try with the latest clearml version 1.13.2?)

  
  
Posted 10 months ago

Oh, I misunderstood then docs/examples, sorry. I'm using pytorch-ignite.

Thanks for the tip!

  
  
Posted 10 months ago

Hi @<1631102016807768064:profile|ZanySealion18>

ClearML doesn't pick up model checkpoints automatically.

What's the framework you are using?
BTW:

Task.add_requirements("requirements.txt")

if you want to specify Just your requirements.txt, do not use add_requirements use:

Task.force_requirements_env_freeze(requirements_file="requirements.txt")

(add requirements with a filename does the same thing, but this is more readable)

  
  
Posted 10 months ago

@<1523701087100473344:profile|SuccessfulKoala55> Kind reminder again, thanks and sorry!

  
  
Posted 10 months ago

@<1523701087100473344:profile|SuccessfulKoala55> kind reminder not to miss this when you catch time, thanks!

  
  
Posted 10 months ago

No worries, sorry for pinging, was just making sure you (or anyone else who might help) doesn't miss it 🙂
I use Task.add_requirements("requirements.txt") right before the Task.init.
In main, I parse arguments command-line, add_requirements, initialize Task and call execute_remotely. After that it's all pretty much the usual workflow. Initialize the model, setup dataloaders, optimizer and run the training. I'm using pytorch-ignite and have model checkpoint made on validation evaluator COMPLETED event.

  
  
Posted 11 months ago

Sorry for the delay 🙏 - how do you import your packages and where do you initialize ClearML relative to the rest of the code?

  
  
Posted 11 months ago

Kind ping on this thread, thanks! 🙂

  
  
Posted 11 months ago

clearml-1.13.1

Task.add_requirements("requirements.txt")
task = Task.init(project_name="My project", task_name="My task")
task.execute_remotely(queue_name="default")
...
  
  
Posted 11 months ago

Hi @<1631102016807768064:profile|ZanySealion18> , can you provide more info on what framework you're using, which ClearML SDK version and how you're initializing the ClearML task?

  
  
Posted 11 months ago

    model_checkpoint = ModelCheckpoint(
        "checkpoint",
        n_saved=2,
        filename_prefix="best",
        score_function=score_function,
        score_name="accuracy",
        global_step_transform=global_step_from_engine(trainer),
    )

    # Save the model after every epoch of val_evaluator is completed
    val_evaluator.add_event_handler(
        Events.COMPLETED, model_checkpoint, {"model": model}
    )
  
  
Posted 11 months ago
839 Views
11 Answers
11 months ago
10 months ago
Tags