Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Is There A Way How I Can Get How Many Minutes The Gpu Has Been Used In A Month? The Duration Of An Iteration Is For Every Run Different If You Vary Batch Size. Model, Or Other Stuff. I Want To Do A Crude Energy Consumption Calculation By Doing A Sum Over

Is there a way how I can get how many minutes the GPU has been used in a month? The duration of an iteration is for every run different if you vary batch size. model, or other stuff.

I want to do a crude energy consumption calculation by doing a sum over all durations of the experiments in that month, and multiply it with the average GPU utilization during training.

There is a total COMPUTE TIME per project, but I haven't been able to find wall clock run times anywhere else (the closest is the start and end time, but I'm using notebooks, so it's not necessarily it was training all the time. Getting this would already be a huge help though).

  
  
Posted 3 years ago
Votes Newest

Answers 4


You can query the system and get all the experiments based on date, then grab the machine GPU metrics.
DefeatedCrab47 check the cleanup service, it queries the system with the Apiclient.
https://github.com/allegroai/trains/blob/10ec4d56fb4a1f933128b35d68c727189310aae8/examples/services/cleanup/cleanup_service.py#L72

  
  
Posted 3 years ago

AgitatedDove14 Thank you, this code example is very helpful!

  
  
Posted 3 years ago

Hi DefeatedCrab47
You mean by trains-agent, or accumulated over all experiences ?

  
  
Posted 3 years ago

Hi AgitatedDove14
Not using trains-agent yet. Just using PyTorch Lightning in Jupyter Notebook with as Logger Trains.
So I'm talking about runtime and GPU usage in experiments.

  
  
Posted 3 years ago
681 Views
4 Answers
3 years ago
one year ago
Tags