Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All - I Am Using The Experiment Tracker And I See That Scalars View Is Not Updated After Some Time? In The Console View I See That My Training Job Terminated At Step 708000, Having Logged Scalar Metrics Every 4 Steps, However In The Scalars Pane I See

Hi all - I am using the experiment tracker and I see that scalars view is not updated after some time? In the console view I see that my training job terminated at step 708000, having logged scalar metrics every 4 steps, however in the scalars pane I see nothing logged after step 101152 🤔 I see this behavior consistently, maybe there is some buffering of scalar values going wrong? I am thinking that I will try to log only every n minutes and see if that mitigates the issue, just wanted to hear if anyone had similar experiences and possible fixes/workarounds?

  
  
Posted 7 months ago
pic
More
StickyShrimp60
0 × 1
Report
Votes Newest

Answers 6

So early/mid experiment run everything aligned fine?

  
  
Posted 7 months ago
pic
More
CostlyOstrich36
0
Report

Hi @<1655744373268156416:profile|StickyShrimp60> , do you have any code that can reproduce this behavior?

  
  
Posted 7 months ago
pic
More
CostlyOstrich36
0
Report

How are the metrics being reported? directly via the Logger module or via automatical logging of some framework? Also how are iterations reported

  
  
Posted 7 months ago
pic
More
CostlyOstrich36
0
Report

Yes, early in the experiment everything seems fine, then updating begins lagging and coming in chucks before not updating at all anymore

  
  
Posted 7 months ago
pic
More
StickyShrimp60
0 × 1
Report

Hi @<1523701070390366208:profile|CostlyOstrich36> - unfortunately not something that I can easily share I am afraid. I am using the hosted solution btw and I really believe I am just logging way too often, I think about once a second for some 12 hours..

  
  
Posted 7 months ago
pic
More
StickyShrimp60
0 × 1
Report

I am logging directly via the Logger module using the report_scalar method, adding iterations parameter in every call. In my recent runs I am only logging every 30s and this does seem to remove the issue do not see the issue

  
  
Posted 7 months ago
pic
More
StickyShrimp60
0 × 1
Report
Write your answer
442 Views
6 Answers
7 months ago
7 months ago
Tags
clearml