its should logged all in the end as I understand

Hmm let me check the code for a minute

its very weird
the train upload to the bucket the files..
ex:
clearml.Task - INFO - Completed model upload to gs://... clearml.Task - INFO - Finished uploadingi have print for the model is uploaded
also i can see all the files in the bucket as i said (model+ metrics )
the only thing that missing is some plots on the clearml server (app ) when i got to the details of the train i cannot see the matrix confusion for example ( but its exists on the bucket )

i thought the error logged it might related to that

i would like to have it also save on the bucket

oh if this is the casse, you can just change the clearml file server to point to GS bucket, everything will be stored there.
Just change your clearml.conf:
files_server: " "https://github.com/allegroai/clearml/blob/d45ec5d3e2caf1af477b37fcb36a81595fb9759f/docs/clearml.conf#L10

Are you saying that in the UI you do not see "confusion matrix" at all, only on the GS bucket ?

AstonishingRabbit13 so is it working now ?

Hi AgitatedDove14 thanks for the help!
i run it now and in the end the task upload the model for me to the bucket
clearml.Task - INFO - Completed model upload to gs://
but when i check i can see in the bucket only the final model.. do you know how can i save all the logs and all the metric images?

Thanks

yes they are on the clearml-server now

i would like to have it also save on the bucket

1. save space on the clearml server
2. i have the model + all its info in one place on the bucket

i’m using the yolov5 repo
https://github.com/ultralytics/yolov5
its should logged all in the end as I understand

Thanks i can see the files now on the bucket
i saw also error in the end of the training
clearml.storage - ERROR - Failed uploading: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url... (Caused by SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac
and in the clearml server some images in the PLOTS tab are missing..
is there something else in the conf that i should change ?

again thanks a lot for the help!

Hi AstonishingRabbit13

now I’m training yolov5 and i want to save all the info (model and metrics ) with clearml to my bucket..

The easiest thing (assuming you are running YOLOv5 with python train.py is to add the following env variable:
CLEARML_DEFAULT_OUTPUT_URI=" " python train.pyNotice that you need to pass your GS credentials here:
https://github.com/allegroai/clearml/blob/d45ec5d3e2caf1af477b37fcb36a81595fb9759f/docs/clearml.conf#L113

Could it be you have some custom SSL certificate installed, or policy ?
can you get other https sites? (for example your clearml-server)

And you are seeing a bunch of the GS SSL errors?

in the bucket i can see all the files now!
but on the clearml server when i go into the train some of the plots are missing..
like Confusion Matrix ..

for now i think i’m ok
the scalars seems to be right and the metrics there is whats import for me..

the error for uploading is weird

again thanks for the help!!

my gcp instance
try to upgrade openssl.. still got error

The confusion matrix shows under debug sample, but the image is empty, is that correct?

the only thing that missing is some plots on the clearml server (app ) when i got to the details of the train i cannot see the matrix confusion for example ( but its exists on the bucket )

How do you report the "matrix confusion" ? (I might have an idea on what's the difference)

on the clearml server i can see only :
F1-Confidence Curve, Precision-Confidence Curve, Precision-Recall Curve, Recall-Confidence Curve,but on the bucket all the rest:

i had that.. in the end the files where uploaded
just some where missing in the clearml server..

is there something else in the conf that i should change ?

I'm assuming the google credentials?
https://github.com/allegroai/clearml/blob/d45ec5d3e2caf1af477b37fcb36a81595fb9759f/docs/clearml.conf#L113

yes..
when i change the files_server to be back the clearml server ( save locally ) i can see everything

do you know how can i save all the logs and all the metric images?

These are stored into clearml-server, no? what am I missing ?

no..
just the error line i mention

This is very odd, can you also put here the file names? maybe an odd character is causing it?
Can you also test it with the latest clearml version (1.8.0) ?

AstonishingRabbit13
https://github.com/googleapis/google-cloud-python/issues/4941#issuecomment-369472576
check the openssl and the date, this seems like SSL low level error (even before authentication)

yes

the error for uploading is weird

wait, are you still getting this error?

(Caused by SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record macWhere is the code running (agent) GCP instance ? your machine ?