Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Guys, I Have Many Questions To Ask, Sorry If This Questions Were Posted Already - If The Answer Exist, Please, Point Me To It. Thank You For Your Help. I'M Training Object Detection Model Using Tf 2.3 Object Detection Api And Use Clearml On Local Serve

Hi guys, I have many questions to ask, sorry if this questions were posted already - if the answer exist, please, point me to it. Thank you for your help.
I'm training object detection model using TF 2.3 Object Detection API and use ClearML on local server (1 GPU). What I need:

  1. Run several tasks sequentually . I've run one task (I've inserted ClearML code in model_main_tf2.py and ran it in command window), cloned it and marked as pending to be executed next in the queue. But after 1st task finished, the second task didn't started - i had to start is manually from command window. How can I make the new task start automatically?
  2. Accuracy and losses are not logged. So far I understand, these params should be logged automatically. The only parameters logged are gpu/cpu/memory load. What's wrong?
  3. I can't see learning rate/optimizer/IOU thresholds, etc. (other params, which can be set in model config file) among hyperparams in task->configuration->hyperparameters menu. Also, how can I set these hyperparameters for optimization using Optuna?
  4. In the following article: https://medium.com/swlh/construction-feat-tf2-object-detection-api-4465a3937c87 data management tool and annotation preview is mentioned. Are these proprietary developed tools for ClearML, or data management and annotation preview can be also found in ClearML?

Thanks.

  
  
Posted 3 years ago
Votes Newest

Answers 33


let me know if you need any help/ have issues trying to reproduce...thanks!

  
  
Posted 3 years ago

@ https://app.slack.com/team/U01J3C692M8 where you able to come up with a solution?

  
  
Posted 3 years ago

And no scalars ?

  
  
Posted 3 years ago

MagnificentSeaurchin79 no need for the detection api (yes definitely a mess to setup), it will be more helpful to get a toy example.

  
  
Posted 3 years ago

BTW MagnificentSeaurchin79 just making sure here:

but I don't see the loss plot in scalars

This is only with Detect API ?

  
  
Posted 3 years ago

clearml == 0.17.5rc5
google_cloud_storage == 1.36.1
joblib == 1.0.1
matplotlib == 3.3.4
numpy == 1.20.0
object_detection == 0.1
opencv_python_headless == 4.5.1.48
pandas == 1.2.3
scikit_learn == 0.24.1
tensorflow == 2.4.0

  
  
Posted 3 years ago

This is only with Detect API ?

I only tested it with the Detect API

  
  
Posted 3 years ago

MagnificentSeaurchin79 making sure the basics work.
Can you see the 3D plots under the Plot section ?
Regrading the Tensors, could you provide a toy example for us to test ?

  
  
Posted 3 years ago

I'll need to see how to extract only the part that we care about

  
  
Posted 3 years ago

It's kind of a pain to setup Tensorflow Object Detection API

  
  
Posted 3 years ago

Thanks!

  
  
Posted 3 years ago

This should have worked with the latest clearml RC.
And you verified it is not working?

  
  
Posted 3 years ago

ping AgitatedDove14 ... 😄

  
  
Posted 3 years ago

Thanks MagnificentSeaurchin79 !
Let me check what's the status with this one, could it be the same as this one?
https://github.com/allegroai/clearml/issues/322

  
  
Posted 3 years ago

oh I see..good luck!

  
  
Posted 3 years ago

image

  
  
Posted 3 years ago

whereas this is what is being logged in your toy example: tf.Tensor(1742.0144, shape=(), dtype=float32)

  
  
Posted 3 years ago

<tf.Tensor 'Loss/RPNLoss/localization_loss:0' shape=() dtype=float32
This is what is being logged as scalar in the OD API

  
  
Posted 3 years ago

yes, that works..but wasn't the issue with logging tensors?

  
  
Posted 3 years ago

image

  
  
Posted 3 years ago

I don't think so..I had a problem before 0.17.5

  
  
Posted 3 years ago

Thanks!

  
  
Posted 3 years ago

Sure!

  
  
Posted 3 years ago

these?

  
  
Posted 3 years ago

only these

  
  
Posted 3 years ago

Hi AgitatedDove14 ! Were you able to reproduce this?

  
  
Posted 3 years ago

Hi MagnificentSeaurchin79
This sounds like a deeper bug (of a sort), I think the best approach is to open a GitHub issue with some code that can reproduce this behavior, or at least enough information so that we could try to catch the bug.
This way we will make sure it is not forgotten.
Sounds good ?

  
  
Posted 3 years ago

Hi AgitatedDove14 , can you please give me some directions on how to follow this?

  
  
Posted 3 years ago

Hi MagnificentSeaurchin79
Could you test with the tesnorflow toy example?
https://github.com/allegroai/clearml/blob/master/examples/frameworks/tensorflow/tensorboard_toy.py

  
  
Posted 3 years ago

This looks strange that only a single scalar is reported.

  
  
Posted 3 years ago
19K Views
33 Answers
3 years ago
8 months ago
Tags