Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
PanickyBee11
Moderator
2 Questions, 6 Answers
  Active since 17 April 2023
  Last activity one year ago

Reputation

0

Badges 1

6 × Eureka!
0 Votes
3 Answers
945 Views
0 Votes 3 Answers 945 Views
Is It Possible To Run In Offline Mode And Still Save The Machine Monitoring Metrics? By Default It Is Monitored For Me In Online Mode But Not In Offline Mode.
Is it possible to run in offline mode and still save the machine monitoring metrics? By default it is monitored for me in online mode but not in offline mode.
one year ago
0 Votes
9 Answers
1K Views
0 Votes 9 Answers 1K Views
Hi, I’M Training On Multi-Node, Clearml Captures Only A Single Machine Utility (Memory/Cpu/Etc.). I Assume It Captures Node 0. Is There A Way To Make It Report All Nodes?
Hi, I’m training on multi-node, clearml captures only a single machine utility (memory/cpu/etc.). I assume it captures node 0. Is there a way to make it repo...
one year ago
0 Hi, I’M Training On Multi-Node, Clearml Captures Only A Single Machine Utility (Memory/Cpu/Etc.). I Assume It Captures Node 0. Is There A Way To Make It Report All Nodes?

@<1523701205467926528:profile|AgitatedDove14> yes & yes, multiple machines and reporting to the same task.

one year ago
0 Hi, I’M Training On Multi-Node, Clearml Captures Only A Single Machine Utility (Memory/Cpu/Etc.). I Assume It Captures Node 0. Is There A Way To Make It Report All Nodes?

pytorch ddp @<1523701070390366208:profile|CostlyOstrich36>

one year ago
0 Hi, I’M Training On Multi-Node, Clearml Captures Only A Single Machine Utility (Memory/Cpu/Etc.). I Assume It Captures Node 0. Is There A Way To Make It Report All Nodes?

image

one year ago
0 Hi, I’M Training On Multi-Node, Clearml Captures Only A Single Machine Utility (Memory/Cpu/Etc.). I Assume It Captures Node 0. Is There A Way To Make It Report All Nodes?

It's launched with torchrun https://pytorch.org/docs/stable/elastic/run.html

I think prefix would be great. It can also make it easier for reporting scalars in general (save the users the need to manually add the rank label). It can also be great to support adding the average of all nodes at the UI level, currently we need a barrier to sync all nodes before reporting a scalar which makes it slower.

one year ago
0 Is It Possible To Run In Offline Mode And Still Save The Machine Monitoring Metrics? By Default It Is Monitored For Me In Online Mode But Not In Offline Mode.

I do see these metrics in the json file, but they are not shown in the UI

one year ago
0 Hi, I’M Training On Multi-Node, Clearml Captures Only A Single Machine Utility (Memory/Cpu/Etc.). I Assume It Captures Node 0. Is There A Way To Make It Report All Nodes?

Will give it a try tomorrow, thanks!

one year ago