I don't think datasets don't have visualization out of the box, you need to add these previews manually. Only HyperDatasets feature from the Scale & Enterprise versions truely visualizes all the data.
According to your code snippet there isn't any visualization add on top of the dataset
Actually, datasets should have an automatic preview...
This is how I usually add visualization
#Report data preview
ds.get_logger().report_table(title="Data Sample", series="First Ten Rows", table_plot=data1[:10])
ds.upload()
ds.finalize()
What if you add images to the dataset? Can you see them being previewed? @<1523701168822292480:profile|ExuberantBat52>
@<1523701070390366208:profile|CostlyOstrich36> It looks like this:
` from clearml import Dataset
IMG_PATH = "/home/mfb/Temp/sample-ds/50-ok.jpg"
Create dataset and add sample image
ds = Dataset.create(dataset_name="Test", dataset_project="Dataset-Test")
ds.add_files(path=IMG_PATH)
ds.upload()
Add and report image
logger = ds.get_logger()
logger.report_image("image", "sample image", iteration=0, local_path=IMG_PATH)
logger.flush()
Finalize the dataset
ds.finalize() `
How about when you view it in the datasets view? Also what version of clearml
package do you have?
CostlyOstrich36 Do you have any idea how I could debug that?
Above is the response for the events.debug_images
CostlyOstrich36 I have used ds.get_logger().report_image()
in a loop in order to upload all the images from the dataset as previews. Then I called flush()
, which returned True
. However, the images are still not shown. Any ideas - how could I debug that?
It isn't a bug, you have to add the previews manually through reporting. For example:
ds = Dataset.create(...) ds.add_files(...) ds.get_logger().report_media(...)
I am using the latest version clearml server
and I am using version 1.9.1
for the sdk.
Here is the code that I am currently using:
if __name__ == "__main__":
# create clearml data processing task
dataset = Dataset.create(
dataset_name="palmer_penguins",
dataset_project="palmer penguins",
dataset_tags=["raw"]
)
dataset_path = "data/raw/penguins.csv"
# add the downloaded files to the current dataset
dataset.add_files(path=dataset_path)
# upload data to clearml server
dataset.upload(verbose=True)
# close the dataset
dataset.finalize(verbose=True)
This is a simple dummy example that I use for testing deployments of clearml server
Here is the output that I would expect:
@<1523701168822292480:profile|ExuberantBat52> Do you have pandas installed on your machine?
CostlyOstrich36 I do see the new "image" in the metric drop down, so something has changed - but there are no preview images there.
Thanks a lot, I will have a look at that!
Might need to refresh page after opening dev tools 🙂
ProudElephant77 , can you please add a code snippet of what you did?
CostlyOstrich36 Unfortunately that didn't helped 😞
This is strange. Exactly same code works for me. What version of clearml are you using? What version is your server?
CostlyOstrich36 WebApp: 1.9.1-312 • Server: 1.9.1-312 • API: 2.23
Also when in this view, open developer tools (F12) and see what calls you get back for debug samples
CostlyOstrich36 The latest clearml package, installed these days - 1.9.0
The dataset view is also empty:
@<1523701070390366208:profile|CostlyOstrich36> Hi John, I have posted that in the screenshot above. Isn't that what you meant?
@<1523701168822292480:profile|ExuberantBat52> , did you add debug samples in a similar fashion? What version of the clearml
sdk are you using? Also what server?
Yes, I am using a virtualenv that has pandas and clearml installed.
@<1523701070390366208:profile|CostlyOstrich36> I am facing the same issue:
{"meta":{"id":"90841d05dfb1431a8d9dfc6bfdb39f9e","trx":"90841d05dfb1431a8d9dfc6bfdb39f9e","endpoint":{"name":"events.debug_images","requested_version":"2.23","actual_version":"2.7"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":"","error_data":{}},"data":{"metrics":[]}}
I am currently running the scripts on WSL ubuntu
The above output is on the clearml community server