Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey Guys, In A Fresh Self-Hosted Clearml Server I Have The Issue That I Cannot See Any Preview Images Of The Files In A Dataset. It Seems Like A Bug, But Maybe I Am Doing Something Wrong?

Hey guys, in a fresh self-hosted ClearML server I have the issue that I cannot see any preview images of the files in a dataset. It seems like a bug, but maybe I am doing something wrong? https://github.com/allegroai/clearml/issues/879 - does someone have an idea what I might be doing wrong?

  
  
Posted 2 years ago
Votes Newest

Answers 42


Can you share a screenshot of what you see?

  
  
Posted 2 years ago

ProudElephant77 , I think you might need to finalize the dataset for it to appear

  
  
Posted 2 years ago

CostlyOstrich36 Hi John, I have posted that in the screenshot above. Isn't that what you meant?

  
  
Posted 2 years ago

Thanks a lot, I will have a look at that!

  
  
Posted 2 years ago

It isn't a bug, you have to add the previews manually through reporting. For example:

ds = Dataset.create(...) ds.add_files(...) ds.get_logger().report_media(...)

  
  
Posted 2 years ago

I am using the latest version clearml server and I am using version 1.9.1 for the sdk.

Here is the code that I am currently using:

if __name__ == "__main__":
    # create clearml data processing task
    dataset = Dataset.create(
        dataset_name="palmer_penguins",
        dataset_project="palmer penguins",
        dataset_tags=["raw"]
    )
    dataset_path = "data/raw/penguins.csv"
    # add the downloaded files to the current dataset
    dataset.add_files(path=dataset_path)
    # upload data to clearml server
    dataset.upload(verbose=True)
    # close the dataset
    dataset.finalize(verbose=True)

This is a simple dummy example that I use for testing deployments of clearml server

Here is the output that I would expect:
image

  
  
Posted 2 years ago

The thing is, even on the community server, not all the datasets have automatic previews. So for the same code/dataset, some of the runs have previews and some of them don't.

  
  
Posted 2 years ago

ExuberantBat52 Do you have pandas installed on your machine?

  
  
Posted 2 years ago

Actually, datasets should have an automatic preview...

  
  
Posted 2 years ago

I am currently running the scripts on WSL ubuntu

  
  
Posted 2 years ago

What if you add images to the dataset? Can you see them being previewed? ExuberantBat52

  
  
Posted 2 years ago

CostlyOstrich36

  
  
Posted 2 years ago

My server is hosted on AWS Fargate

  
  
Posted 2 years ago

CostlyOstrich36 I have used ds.get_logger().report_image() in a loop in order to upload all the images from the dataset as previews. Then I called flush() , which returned True . However, the images are still not shown. Any ideas - how could I debug that?

  
  
Posted 2 years ago

CostlyOstrich36 The latest clearml package, installed these days - 1.9.0

The dataset view is also empty:

  
  
Posted 2 years ago

CostlyOstrich36 Do you have any idea how I could debug that?

  
  
Posted 2 years ago

I don't think datasets don't have visualization out of the box, you need to add these previews manually. Only HyperDatasets feature from the Scale & Enterprise versions truely visualizes all the data.

According to your code snippet there isn't any visualization add on top of the dataset

  
  
Posted 2 years ago

CostlyOstrich36 It looks like this:
image

  
  
Posted 2 years ago

Yes, I am using a virtualenv that has pandas and clearml installed.

  
  
Posted 2 years ago

Can you please paste the response from events.debug_images ?

  
  
Posted 2 years ago

Also when in this view, open developer tools (F12) and see what calls you get back for debug samples

  
  
Posted 2 years ago

ExuberantBat52 , did you add debug samples in a similar fashion? What version of the clearml sdk are you using? Also what server?

  
  
Posted 2 years ago

Please check what you get for events.debug_images in network section of developer tools (F12) when trying to view the preview in the dataset

  
  
Posted 2 years ago

This is strange. Exactly same code works for me. What version of clearml are you using? What version is your server?

  
  
Posted 2 years ago

How about when you view it in the datasets view? Also what version of clearml package do you have?

  
  
Posted 2 years ago

Hmm my bad, I wasn't aware of this 🙂

  
  
Posted 2 years ago

So I added the snippet above to the code,
and now the preview for the first 10 rows shows up. However, the automatic preview is still not working.

  
  
Posted 2 years ago

Might need to refresh page after opening dev tools 🙂

  
  
Posted 2 years ago

This is how I usually add visualization

#Report data preview
ds.get_logger().report_table(title="Data Sample", series="First Ten Rows", table_plot=data1[:10])
ds.upload()
ds.finalize()
  
  
Posted 2 years ago

Above is the response for the events.debug_images

  
  
Posted 2 years ago