Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello Everyone, I Have An Video Classification Model. I Have A Lot Of Metadata About The Videos In A Csv File, I First Preprocess The Metadata And In Pandas Df. (There May Be Lost Videos, Invalid Videos Etc, So I Remove Those From The Df). I Want To Log T

Hello Everyone, I have an video classification model. I have a lot of metadata about the videos in a csv file, I first preprocess the metadata and in pandas df. (There may be lost videos, invalid videos etc, so I remove those from the df). I want to log the last clean df statistics into clearml (Something like StatisticsGen of TFX). How can I log my data statistics?

Thank you in advance.

  
  
Posted 5 months ago
Votes Newest

Answers 19


Oh, I think that is for a very small data. I don't think it works for me.

  
  
Posted 5 months ago

I mean, depend on what do you want to report ... if you want to stick to table, I suggest earlier to gather your stats in table format ...
Otherwise, matplotlib seems to be the most user friendly way

  
  
Posted 5 months ago

I know. And the very fast help 🙏 😀

  
  
Posted 5 months ago

Can I see visualization of for example categorical columns as bar graphs?

  
  
Posted 5 months ago

Pefect, thanks. I will take a look at that.

  
  
Posted 5 months ago

and just came across this: None
That sounds like what you may be looking for

  
  
Posted 5 months ago

None

  
  
Posted 5 months ago

yup, you have the flexibility and option, that what so nice with ClearML

  
  
Posted 5 months ago

because, the param name that takes in the df is table_plot

  
  
Posted 5 months ago

I also use this: None
Which can give more control

  
  
Posted 5 months ago

you can upload the df as artifact.
Or the statistics as a DataFrame and upload as artifact ?

  
  
Posted 5 months ago

if you want plot, you can simply generate plot with matplotlib and clearml can upload them in the Plot or Debug Sample section

  
  
Posted 5 months ago

Yeah actually you are right. I can report the stats as a table, not the whole data.

  
  
Posted 5 months ago

At least I can do that along with matplotlib

  
  
Posted 5 months ago

never tried 😛

  
  
Posted 5 months ago

No worries. Thank you so much.

  
  
Posted 5 months ago

with

    df = pd.DataFrame({'num_legs': [2, 4, 8, 0],
                       'num_wings': [2, 0, 0, 0],
                       'num_specimen_seen': [10, 2, 1, 8]},
                      index=['falcon', 'dog', 'spider', 'fish'])

    import clearml
    task = clearml.Task.current_task()
    task.get_logger().report_table(title='table example', series='pandas DataFrame', iteration=0, table_plot=df)
    # logger.report_table(title='table example',series='pandas DataFrame',iteration=0,table_plot=df)
    task.close()

image

  
  
Posted 5 months ago

report_table seems to be the most straight forward without matplotlib integration. Do you think it has plotting features?

  
  
Posted 5 months ago

I will have to go for matplotlib or seaborn options.

  
  
Posted 5 months ago
491 Views
19 Answers
5 months ago
5 months ago
Tags