Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello. I'M Running Some Pytorch-Ignite Code Inside A Docker Clearml-Agent. At Some Point, I Have A Checkpoint That Should Upload Information From Inside The Docker Container To My Minio Server. But I Get A Broken Pipe.

Hello. I'm running some pytorch-ignite code inside a docker clearml-agent. At some point, i have a checkpoint that should upload information from inside the docker container to my minio server. But i get a broken pipe. 2022-05-17 08:33:06,296 Trainer INFO: checkpoint filename : train_vgg11_checkpoint_3_MeanF1= http://0.6860.pt , priority : 0.6859513018468952
2022-05-17 08:33:07,409 Trainer INFO: uploaded filename : /app/.clearml/venvs-builds/3.8/task_repository/rocknet.ignite.git/rocknet/experiments/2022_05_17_08_28_45_j2fy2pk3/checkpoint/clearml-cache/train_vgg11_r2_0.6860.pt
2022-05-17 08:33:09,288 - clearml.storage - INFO - Starting upload: /tmp/.clearml.upload_model_249ygzk8.tmp => http://minio-console.10.68.0.250.nip.io:80/RockNet/Clone Of MidSize.4b1e2cec92cc4a9dbc8f7bbc6c0f29fc/models/train_vgg11_r2_0.6860.pt
2022-05-17 08:33:09,289 Trainer INFO: upload artifacts to clearml file server...
2022-05-17 08:33:09,289 Trainer INFO: checkpoint filename : valid_vgg11_checkpoint_3_MeanF1= http://0.6002.pt , priority : 0.6001910604748011
2022-05-17 10:33:14
2022-05-17 08:33:10,411 - clearml.storage - ERROR - Exception encountered while uploading ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

  
  
Posted 2 years ago
pic
More
MotionlessCoral18
0 × 1
Report
Votes Newest

Answers 7

Hi MotionlessCoral18 , when you're using an external storage solution (such as minio), it's only the ClearML SDK client working against minio, it will not involve the clearml fileserver

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

To your point of view, it may be related with the sdk client that triggers the upload ? with urllib request ?

  
  
Posted 2 years ago
pic
More
MotionlessCoral18
0 × 1
Report

Yes, i even got a "upload finished" message et the whole process goes to end.

  
  
Posted 2 years ago
pic
More
MotionlessCoral18
0 × 1
Report

As far i know, a server get a SIG_PIPE event on a socket when a client died too soon or is closed by user, but i don't know who get the broken pipe ? Is it the clearml file server ( who masters the upload, i guess) ? Is it due to my minio server ? Who is the client that died before upload is finished ?

  
  
Posted 2 years ago
pic
More
MotionlessCoral18
0 × 1
Report

Does you clearml Task keep running after this error?

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

It's actually using boto3

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

only a "upload failed" and no data in my S3 bucket

  
  
Posted 2 years ago
pic
More
MotionlessCoral18
0 × 1
Report
Write your answer
783 Views
7 Answers
2 years ago
one year ago
Tags
clearml