Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey! Trains Looks Super Promising For Our Project, But I Struggle To Understand The Saving And Model Aspect Of Trains. I Have A Pytorch Model That I Save With Some Additional Data Every Epoch And I Would Like To Integrate That With Trains. Is That Possibl

Hey! Trains looks super promising for our project, but I struggle to understand the saving and model aspect of trains. I have a pytorch model that I save with some additional data every epoch and I would like to integrate that with trains. Is that possible? I tried the code snippet from https://allegro.ai/docs/tutorials/tutorial_explicit_reporting/ , but I can't find an explicit function for saving snapshots with trains and when I did perform torch.save, nothing updated in the directory. Furthermore, the folder structure trains creates is a little odd. I run the following code:

` model_snapshots_path = "trains_storage/"
if not os.path.exists(model_snapshots_path):
os.makedirs(model_snapshots_path)

task = Task.init(
project_name=logging_config["project_name"],
task_name=config.experiment_name,
output_uri=model_snapshots_path) `And it creates the folder structure: trains_storage/trains_storage
Inside the project folder.

I assume I am not suppose to explicitly give the path to trains_storage when I do torch.save because the documentation says it will generate the structure on its own. Which makes me think I am probably wrong in thinking trains somehow picks up that I am saving the model with torch.

  
  
Posted 4 years ago
Votes Newest

Answers 7


Oh sorry, I forgot to specify one thing. I only really want to save once (because the model is large), so I hoped the model would only be saved to the Trains URI. This was why I was wondering if there was an explicit way to call trains to save, so I don't also save at the location given in torch.save(...). Is this possible?

  
  
Posted 4 years ago

Hi, sorry for late reply! Yes that could work in a way, though I think only saving to the model snapshot feature would have been great!

  
  
Posted 4 years ago

I see, I will try this code snippet 🙂

  
  
Posted 4 years ago

Hi! So if I understand the documentation correctly, then TRAINS support saving snapshots to some uri. Just to test it for now, I just want it to save the snapshots locally into trains_storage with the file structure given in the documentation is referenced previously.

I think the first thing I should ask is just how do I save a pytorch snapshot with trains? Can it be done explicitly with a method, or is it done implicitly by TRAINS somehow picking up that I called torch.save(...)?

What I hope to achieve is: Save a snapshot of my model for each epoch using TRAINS to the given uri

  
  
Posted 4 years ago

Trains do patch the torch save function 🙂

If you like, you can save the model for each epoch by having a unique name for it. The model will be saved in the output_uri path you have in the Task.init command.
For example, this code will save a model for every epoch:

for epoch in range(num_of_epoch): # Create a model torch.save(model, "model_epoch_{}".format(epoch))

  
  
Posted 4 years ago

Hi UnsightlyShark53 ,

Trying to understand the scenario, so you want the model to be saved in trains_storage dir but trains saves it in trains_storage/trains_storage ? Or the torch.save doesn't save in the path?

  
  
Posted 4 years ago

Hi UnsightlyShark53 ,

If you want to save anything from your experiment, you can use the artifacts, you can view this example - https://github.com/allegroai/trains/blob/master/examples/artifacts_toy.py

Can this do the trick?

  
  
Posted 4 years ago
653 Views
7 Answers
4 years ago
one year ago
Tags
Similar posts