Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi There! I'Ve Encountered Some Strange Behaviour With A Clearml Server Self Hosted Via Aws Ec2 Ami (Webapp: 1.14.1-448 • Server: 1.14.1-448 • Api: 2.28). For Some Reason It Doesn'T Capture Tensorboard Logs And

Hi there! I've encountered some strange behaviour with a ClearML server self hosted via AWS EC2 AMI (WebApp: 1.14.1-448 • Server: 1.14.1-448 • API: 2.28). For some reason it doesn't capture TensorBoard logs and scalars and debug samples tabs remain empty, while if I test the same code with the ClearML server (WebApp: 1.14.0-431 • Server: 1.14.0-431 • API: 2.28) hosted on the same machine where I run the code (or on another local machine within our VPN) it works as expected. So, could it be that I missed something in the configuration of a server on AWS? I've checked clearml.conf in both cases and the only difference is the addresses (and credentials ofc) and the fact that AWS server is configured to use subdomains and https connection.

  
  
Posted 7 months ago
Votes Newest

Answers 5


@<1523701087100473344:profile|SuccessfulKoala55> very sorry to bother you via the mention! Just wanted to ask if you (or probably somebody else from the team) have any ideas what could be the reason of missing scalars here?

  
  
Posted 7 months ago

I've also tried to run 2 reporting examples in case of AWS hosted server:

  • None - in this case every plot was there as it should be, in the web UI under the scalars tab (including :monitor:machine with info on iterations, cpu usage, etc.)
  • None - with this sample scalars tab is completely empty as well as a debug samples
  
  
Posted 7 months ago

Thank you for a suggestion! These are the logs I've collected after running this sample None And just for the context, I can see console logs under the console tab, output models as well as a git diff uploaded to artifacts , but no scalars

  
  
Posted 7 months ago

Apparently, the main cause was a big git diff file (about 10Mb because of the Jupyter Notebook), which somehow prevented any other upload to the server (not sure why, probably there is a timeout on any upload operation). I would much appreciate if somebody know or can guess why git diff was causing such an issue

  
  
Posted 7 months ago

Hi @<1684010629741940736:profile|NonsensicalSparrow35> , there's actually no change in that regard in the version, so I can't think of any reason. Can you attach the ES and apiserver logs from the version that doesn't work for you? perhaps there are some errors there?

  
  
Posted 7 months ago