Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey, I Run A Programm Without Allegro On The Gpu And It Works. Then I Run It With Allegro. But The Training Does Not Start. The Gpu Is Allocated But The Training Does Not Start. The Programm Is Stuck. I Am Using The Newest Allegro Version 1.0.2 How Can I

hey,
I run a programm without allegro on the gpu and it works.
Then I run it with allegro. But the training does not start.
The gpu is allocated but the training does not start.
The programm is stuck.
I am using the newest allegro version 1.0.2
How can I solve this?
After 3 Minutes I get the message:
None of the MLIR optimization passes are enabled (registered 2)
ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start

  
  
Posted 2 years ago
pic
More
UnsightlySeagull42
0 × 1
Report
Votes Newest

Answers 5

thanks.
i tried 1.0.4rc0 but get the same error.
Output from allegro:

2021-06-01 15:51:59.984367: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)

2021-06-01 15:52:00.019168: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3399905000 Hz

2021-06-01 15:52:00.683090: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11

2021-06-01 15:52:01.099961: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11

ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start

I run it for 2 hours but it did not train

  
  
Posted 2 years ago
pic
More
UnsightlySeagull42
0 × 1
Report

Hi UnsightlySeagull42
How can I reproduce this behavior ?
Are you getting all the console logs ?
Is it only the Tensorboard that is missing ?

  
  
Posted 2 years ago
pic
More
AgitatedDove14
0 × 1
Report

Hi UnsightlySeagull42
Could you test with the latest RC
pip install clearml==1.0.4rc0Also could you provide some logs?

  
  
Posted 2 years ago
pic
More
AgitatedDove14
0 × 1
Report

sorry
I solved the mistake. there was a mistake in my file path and then the training could not be started

  
  
Posted 2 years ago
pic
More
UnsightlySeagull42
0 × 1
Report

No worries 🙂

  
  
Posted 2 years ago
pic
More
AgitatedDove14
0 × 1
Report
Write your answer
617 Views
5 Answers
2 years ago
one year ago
Tags
clearml