Hi TenseOstrich47 whats the matplotlib version and clearml version you are using ?
Hangs there ? could it be that it's uploading slowly ?
Can you check the network ?
(with matplotlib 3.2+ I get no warning, let me check with 3.1)
2021-03-01 20:51:55,655 - clearml.Task - INFO - Completed model upload to s3://15gifts-clearml/artefacts/pre-engine-traits/logistic-regression-paths-and-sales-tfidf-device-brand.8d68e9a649824affb9a9edf7bfbe157d/models/tfidf-logistic-regression-1614631915-8d68e9a649824affb9a9edf7bfbe157d.pkl *****
2021-03-01 20:52:01
2021-03-01 20:51:57,207 - clearml.Task - INFO - Waiting to finish uploads
it just hangs when trying to upload. maybe that is the reason that the plots are not logging?
/home/ubuntu/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/clearml/utilities/plotlympl/mpltools.py:371: MatplotlibDeprecationWarning: The is_frame_like function was deprecated in Matplotlib 3.1 and will be removed in 3.3.
This is the last print statement before it hangs
No I mean configure the files_server
in the clearml.conf
` # Plot the confusion matrix for predictions
sns.heatmap(
preds_confusion_percentage, annot=True, fmt=".3f", linewidths=.5,
square=True, cmap='Blues_r'
)
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
title_str = f'Accuracy Score: {round(score, 2)}\n{TRANSFORM_TYPE}'
plt.title(title_str, size=15)
task.logger.report_matplotlib_figure(
title=f"Performance Heatmap - {model_export_name}",
series="Device Brand Predictions",
iteration=0,
figure=plt,
report_image=False,
)
plt.show() `
Just upgraded matplotlib, going to test now
Actually unless you specifically detached the matplotlib automagic, any plt.show() will be automatically reported.
Seems like it is working (including seaborn)
The task is dependent on a few artefacts from another task. Is there anything else I can do here?
its a seaborn heatmap that needs to be plotted. not sure if that is useful at all
I dont think its that. its a 20kb file upload. This was the last message just printedClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-star
Can you try to run the example code, see if that works for you?
So do I set the report_image to False for the plots to appear in the plots tab?
If uploaded as image, what is the target destination for logging?
I will try add some print statements to test the hanging issue
TenseOstrich47 notice:task.logger.report_matplotlib_figure( title=f"Performance Heatmap - {name}", series="Device Brand Predictions", iteration=0, figure=figure, **report_image=True,** )
report_image=True means it will be uploaded as an image not a plot (like imshow), the default is False , which would put it under Plots section
Code you add a few prints, and see where it hangs ? there's no reason for it to hang (even the plot upload is done in the background)
could it be how I am trying to log the figure manually?
I removed it and I still get the same error 😞
Could you see if that makes a difference ?
Hmm could you try to upload to your files server (not the S3)
Maybe some credentials error ?
Yes this seems like it is stuck, could you test with the demo server ?
(basically remove the clearml.conf it will connect automatically)