Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello. I'M Running Some Pytorch-Ignite Code Inside A Docker Clearml-Agent. At Some Point, I Have A Checkpoint That Should Upload Information From Inside The Docker Container To My Minio Server. But I Get A Broken Pipe.

Hello. I'm running some pytorch-ignite code inside a docker clearml-agent. At some point, i have a checkpoint that should upload information from inside the docker container to my minio server. But i get a broken pipe. 2022-05-17 08:33:06,296 Trainer INFO: checkpoint filename : train_vgg11_checkpoint_3_MeanF1= http://0.6860.pt , priority : 0.6859513018468952
2022-05-17 08:33:07,409 Trainer INFO: uploaded filename : /app/.clearml/venvs-builds/3.8/task_repository/rocknet.ignite.git/rocknet/experiments/2022_05_17_08_28_45_j2fy2pk3/checkpoint/clearml-cache/train_vgg11_r2_0.6860.pt
2022-05-17 08:33:09,288 - clearml.storage - INFO - Starting upload: /tmp/.clearml.upload_model_249ygzk8.tmp => http://minio-console.10.68.0.250.nip.io:80/RockNet/Clone Of MidSize.4b1e2cec92cc4a9dbc8f7bbc6c0f29fc/models/train_vgg11_r2_0.6860.pt
2022-05-17 08:33:09,289 Trainer INFO: upload artifacts to clearml file server...
2022-05-17 08:33:09,289 Trainer INFO: checkpoint filename : valid_vgg11_checkpoint_3_MeanF1= http://0.6002.pt , priority : 0.6001910604748011
2022-05-17 10:33:14
2022-05-17 08:33:10,411 - clearml.storage - ERROR - Exception encountered while uploading ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

  
  
Posted one year ago
pic
More
MotionlessCoral18
0 × 1
Report
Votes Newest

Answers 7

It's actually using boto3

  
  
Posted one year ago
pic
More
SuccessfulKoala55
0 × 1
Report

As far i know, a server get a SIG_PIPE event on a socket when a client died too soon or is closed by user, but i don't know who get the broken pipe ? Is it the clearml file server ( who masters the upload, i guess) ? Is it due to my minio server ? Who is the client that died before upload is finished ?

  
  
Posted one year ago
pic
More
MotionlessCoral18
0 × 1
Report

Hi MotionlessCoral18 , when you're using an external storage solution (such as minio), it's only the ClearML SDK client working against minio, it will not involve the clearml fileserver

  
  
Posted one year ago
pic
More
SuccessfulKoala55
0 × 1
Report

To your point of view, it may be related with the sdk client that triggers the upload ? with urllib request ?

  
  
Posted one year ago
pic
More
MotionlessCoral18
0 × 1
Report

Yes, i even got a "upload finished" message et the whole process goes to end.

  
  
Posted one year ago
pic
More
MotionlessCoral18
0 × 1
Report

only a "upload failed" and no data in my S3 bucket

  
  
Posted one year ago
pic
More
MotionlessCoral18
0 × 1
Report

Does you clearml Task keep running after this error?

  
  
Posted one year ago
pic
More
SuccessfulKoala55
0 × 1
Report
Write your answer
628 Views
7 Answers
one year ago
one year ago
Tags
clearml