Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All - I Am Using The Experiment Tracker And I See That Scalars View Is Not Updated After Some Time? In The Console View I See That My Training Job Terminated At Step 708000, Having Logged Scalar Metrics Every 4 Steps, However In The Scalars Pane I See

Hi all - I am using the experiment tracker and I see that scalars view is not updated after some time? In the console view I see that my training job terminated at step 708000, having logged scalar metrics every 4 steps, however in the scalars pane I see nothing logged after step 101152 🤔 I see this behavior consistently, maybe there is some buffering of scalar values going wrong? I am thinking that I will try to log only every n minutes and see if that mitigates the issue, just wanted to hear if anyone had similar experiences and possible fixes/workarounds?

  
  
Posted 24 days ago
Votes Newest

Answers 6


Hi @<1655744373268156416:profile|StickyShrimp60> , do you have any code that can reproduce this behavior?

  
  
Posted 24 days ago

Hi @<1523701070390366208:profile|CostlyOstrich36> - unfortunately not something that I can easily share I am afraid. I am using the hosted solution btw and I really believe I am just logging way too often, I think about once a second for some 12 hours..

  
  
Posted 24 days ago

Yes, early in the experiment everything seems fine, then updating begins lagging and coming in chucks before not updating at all anymore

  
  
Posted 24 days ago

How are the metrics being reported? directly via the Logger module or via automatical logging of some framework? Also how are iterations reported

  
  
Posted 23 days ago

So early/mid experiment run everything aligned fine?

  
  
Posted 24 days ago

I am logging directly via the Logger module using the report_scalar method, adding iterations parameter in every call. In my recent runs I am only logging every 30s and this does seem to remove the issue do not see the issue

  
  
Posted 22 days ago
48 Views
6 Answers
24 days ago
21 days ago
Tags