Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Am Trying To Upload A Model But I Am Getting The Following Error:

Hi, I am trying to upload a model but I am getting the following error:

2021-07-05 17:21:14,625 - clearml.storage - ERROR - Failed uploading: cannot schedule new futures after interpreter shutdown
Code:
` try:
best_model = type(self).load_from_checkpoint(
list(Path(os.getcwd()).glob("best-*"))[0]
)
conv2onnx(best_model, "best_model.onnx", size=(224, 224))
output_model = clearml.OutputModel()
output_model.update_weights(str(Path(os.getcwd()) / "best_model.onnx"))

    except IndexError:
        print("Best model not found, skipping torch save.") `

It is the test_epoch_end method on pytorch lightning. I also tried to upload outside the LightningModule, but I get the same error.

  
  
Posted 3 years ago
Votes Newest

Answers 11


Hi SkinnyPanda43

cannot schedule new futures after interpreter shutdown

This seems like a strange exception...
What's the setup here ? jupyter notebook ? how is the interpreter down ?

  
  
Posted 3 years ago

Weird issue, I'll make sure we fix compatibility with python 3.9

  
  
Posted 3 years ago

So downgrading to python 3.8 would be a workaround?

  
  
Posted 3 years ago

Yes, tried with python 3.8, now it works.

  
  
Posted 3 years ago

I get the same error:

⋊> /d/c/p/c/e/reporting on master ◦ python model_config.py (longoeixo) 17:48:14
ClearML Task: created new task id=xxx
ClearML results page: xxx
Any model stored from this point onwards, will contain both model_config and label_enumeration
2021-07-05 17:48:18,463 - clearml.storage - INFO - Starting upload: my_best_model.bin => xxx/models/my_best_model.bin
2021-07-05 17:48:18,464 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis
2021-07-05 17:48:18,466 - clearml.storage - ERROR - Failed uploading: cannot schedule new futures after interpreter shutdown
2021-07-05 17:48:18,466 - clearml.storage - ERROR - Exception encountered while uploading Upload failed
2021-07-05 17:48:18,466 - clearml.Task - INFO - Failed model upload
2021-07-05 17:48:19,373 - clearml.Task - INFO - Finished repository detection and package analysis

  
  
Posted 3 years ago

This is being started as a command line script.

Also tried saving the model with:
task.set_model_config(cfg.model) task.update_output_model("best_model.onnx")But got the same exception,

  
  
Posted 3 years ago

Its a S3 bucket, it is working since I am able to upload models before this call and also custom artifacts on the same script.
Ubuntu 18.04
Python: 3.9.5
Clearml: 1.0.4

  
  
Posted 3 years ago

What's the output_uri you are passing ?
And the OS / Python version?

  
  
Posted 3 years ago

SkinnyPanda43 issue verified, this seems to be related to python 3.9 and subprocesses.
Let me check what we can do

  
  
Posted 3 years ago
1K Views
11 Answers
3 years ago
one year ago
Tags
Similar posts