Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Have Another Bug To Report For Clearml-Server 1.2 (Self Hosted) In The Console Logs Of An Experiments, I Cannot See The Latest Logs. Eg My Experiment Is Done, But I Can Only See The Logs Of To The Installation Of The Packages. If I Download The Log

Hi, I have another bug to report for clearml-server 1.2 (self hosted)
In the console logs of an experiments, I cannot see the latest logs. Eg my experiment is done, but I can only see the logs of to the installation of the packages. If I download the log file, I can clearly see that there are other, more recent logs missing

  
  
Posted 2 years ago
Votes Newest

Answers 17


However when downloading the log manually it appears all the data is there?

  
  
Posted 2 years ago

CostlyOstrich36 , actually this only happens for a single agent. The weird thing is that I have a machine with two gpus, and I spawn two agents, one per gpus. Both have the same version. For one, I can see all the logs, but not for the other

  
  
Posted 2 years ago

CostlyOstrich36 Were you able to reproduce it? That’s rather annoying 😅

  
  
Posted 2 years ago

Hi CostlyOstrich36 , this weekend I took a look at the diffs with the previous version ( https://github.com/allegroai/clearml-server/compare/1.1.1...1.2.0# ) and I saw several changes related to the scrolling/logging:
apiserver/bll/event/ http://log_events_iterator.py apiserver/bll/event/ http://events_iterator.py apiserver/config/default/services/_mongo.conf apiserver/database/model/ http://base.py apiserver/services/ http://events.pyI suspect that one of these changes might be responsible for the bug I am facing, wdyt?

  
  
Posted 2 years ago

CostlyOstrich36 yes, when I scroll up, a new events.get_task_log is fired and the response doesn’t contain any log (but it should)

  
  
Posted 2 years ago

CostlyOstrich36 I updated both agents to 1.1.2 and still go the same problem unfortunately. Since I can download the full log file from the Web UI, I guess the agents are reporting correctly?
Could it be that the elasticsearch does not return all the requested logs when it is queried from the WebUI to display it in the console?
Now that I think about it, I remember that on the changelog of the clearml-server 1.2.0 the following is listed:
Fix UI Workers & Queues and Experiment Table pages display mismatching experiment runtime values #100Can it be that somehow the logic of querying logs was affected by this change? Ie. the webUI asks the logs with a wrong timestamp, hence not getting all the logs? That would explain the current bug.
Or maybe something introduced with the changes on Reddit scrolling?
To me, it really looks like a bug from the way the WebUI fetches the logs and show them in the console, I got this problem only after upgrading the clearml-server to 1.2.0
I am happy to send you a screen recording of the problem if it helps understanding it better

  
  
Posted 2 years ago

CostlyOstrich36 , this also happens with clearml-agent 1.1.1 on a aws instance…

  
  
Posted 2 years ago

JitteryCoyote63 , if you go to a completed experiment you only see the packages stage installed in the log?
What OS/ClearML-Agent are you running?

  
  
Posted 2 years ago

No, they have different names - I will try to update both agents to the latest versions

  
  
Posted 2 years ago

Hi CostlyOstrich36 , one more observation: it looks like when I don’t open the experiment in the webUI before it is finished, then I get all the logs correctly. It is when I open the experiment in the webUI while it is running that I don’t see all the logs.
So it looks like there is an effect of caching (the logs are retrieved only once, when I open the experiment for the first time), and not afterwards (or rarely). Is that possible?

  
  
Posted 2 years ago

JitteryCoyote63 , if you hit F12 and open the console in the webUI you should see some calls going out called events.get_task_log , can you take a peek and see if the logs are missing from there?

  
  
Posted 2 years ago

Sure, if you can post it here or send in private if you prefer it would be great

  
  
Posted 2 years ago

yes, 1.2 indeed

  
  
Posted 2 years ago

JitteryCoyote63 , doesn't seem to happen to me. I'll try raising a clean server and see if this happens then. You're running with 1.2, correct?

  
  
Posted 2 years ago

I am sorry to give infos that are not very precise, but it’s the best I can do - Is this bug happening only to me?

  
  
Posted 2 years ago

Interesting! Do they happen to have the same machine name in UI?

  
  
Posted 2 years ago

I think it comes from the web UI of the version 1.2.0 of clearml-server, because I didn’t change anything else

  
  
Posted 2 years ago
760 Views
17 Answers
2 years ago
one year ago
Tags
Similar posts