Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All - I Am Using The Experiment Tracker And I See That Scalars View Is Not Updated After Some Time? In The Console View I See That My Training Job Terminated At Step 708000, Having Logged Scalar Metrics Every 4 Steps, However In The Scalars Pane I See

Hi all - I am using the experiment tracker and I see that scalars view is not updated after some time? In the console view I see that my training job terminated at step 708000, having logged scalar metrics every 4 steps, however in the scalars pane I see nothing logged after step 101152 🤔 I see this behavior consistently, maybe there is some buffering of scalar values going wrong? I am thinking that I will try to log only every n minutes and see if that mitigates the issue, just wanted to hear if anyone had similar experiences and possible fixes/workarounds?

  
  
Posted one month ago
pic
More
StickyShrimp60
0 × 1
Report
Votes Newest

Answers 6

I am logging directly via the Logger module using the report_scalar method, adding iterations parameter in every call. In my recent runs I am only logging every 30s and this does seem to remove the issue do not see the issue

  
  
Posted one month ago
pic
More
StickyShrimp60
0 × 1
Report

How are the metrics being reported? directly via the Logger module or via automatical logging of some framework? Also how are iterations reported

  
  
Posted one month ago
pic
More
CostlyOstrich36
0
Report

Yes, early in the experiment everything seems fine, then updating begins lagging and coming in chucks before not updating at all anymore

  
  
Posted one month ago
pic
More
StickyShrimp60
0 × 1
Report

Hi @<1523701070390366208:profile|CostlyOstrich36> - unfortunately not something that I can easily share I am afraid. I am using the hosted solution btw and I really believe I am just logging way too often, I think about once a second for some 12 hours..

  
  
Posted one month ago
pic
More
StickyShrimp60
0 × 1
Report

Hi @<1655744373268156416:profile|StickyShrimp60> , do you have any code that can reproduce this behavior?

  
  
Posted one month ago
pic
More
CostlyOstrich36
0
Report

So early/mid experiment run everything aligned fine?

  
  
Posted one month ago
pic
More
CostlyOstrich36
0
Report
Write your answer
76 Views
6 Answers
one month ago
one month ago
Tags
clearml