AbruptHedgehog21 what exactly do you store as a Mode file ? is this a python object pickled ?
ComfortableShark77 are you saying you need "transformers" in the serving container?
we will try to use Triton, but it’s a bit hard with transformer model.
All extra packages we add in serving)
So it should work, you can also run your preprocess class manually from your own machine (for debugging), if you pass to it a local file (basically the downloaded model file from the UI, it should work
it. But it’s maybe not the best solution
Yes... it is not, separating the pre/post to CPU instance and letting triton do the GPU serving is a lot more efficient than using pytorch vanilla
ohh AbruptHedgehog21 if this is the case, why don't you store the model with
torch.jit.save and use Triton to run the model ?
(BTW: if you want a full custom model serve, in this case you would need to add torch to the list of python packages)
Hey, maybe AgitatedDove14 or ExasperatedCrab78 can help
I try to add it to ClearML Serving, but it call
method by default
If this is the case, then the statement above is odd to me, if this is a custom engine, who exactly is calling "
forward " ?
(in you code example you specifically call generate, as you should)
AgitatedDove14 we will try to use Triton, but it’s a bit hard with transformer model.
All extra packages we add in serving)
We made now like this. We in preprocessor.py load our model from S3 bucket and then use it. But it’s maybe not the best solution
AgitatedDove14 I need to call
generate method of my model, but by default it calls
AgitatedDove14 Hi. I and Vlad work together and I think I can paraphrase his question.
We’ve got out clearml-serving and we trained our model. Then we want to add that model to serving. But we need to write our custom
preprocess.py in which we need to call method generate from our model. But we do not exactly understand how we can load/refer to our model.
In examples about custom engine we’ve got this
class Preprocess(object): """ Notice the execution flows is synchronous as follows: 1. RestAPI(...) -> body: dict 2. preprocess(body: dict, ...) -> data: Any 3. process(data: Any, ...) -> data: Any 4. postprocess(data: Any, ...) -> result: dict 5. RestAPI(result: dict) -> returned request """ def __init__(self): """ Set any initial property on the Task (usually model object) Notice these properties will be accessed from multiple threads. If you need a stateful (per request) data, use thestate
dict argument passed to pre/post/process functions """ # set internal state, this will be called only once. (i.e. not per request) self._model = Noneand the question is what we need to write in
self._model to load our model in serving? Or how we can refer to our model in
AgitatedDove14 we store .pt model. And we need for model inference method
generate . If we want to load model we need
AgitatedDove14 My model has method generate. i would like to call it. How can i get loaded automaticly model from Preprocess object. Preprocess file
` from typing import Any, Callable, Optional
from transformers import TrOCRProcessor
import numpy as np
Notice Preprocess class Must be named "Preprocess"
def __init__(self): self.processor = TrOCRProcessor.from_pretrained("microsoft/trocr-small-printed") def preprocess(self, body: dict, state: dict, collect_custom_statistics_fn=None) -> Any: return self.processor.batch_decode(np.array(body.get("image"))) def process( self, data: Any, state: dict, collect_custom_statistics_fn: Optional[Callable[[dict], None]], ) -> Any: model = #get model data = model.generate(data.pixel_values) return data def postprocess(self, data: Any, state: dict, collect_custom_statistics_fn=None) -> dict: return dict(predict=data.tolist()) `I trained model and log it to ClearML Server. I try to add it to ClearML Serving, but it call ` forward ` method by default
How can i get loaded model in Preporcess class in ClearML Serving?
You mean your preprocess class needs a python package or is it your own module ?
AgitatedDove14 KindChimpanzee37 Can you help with this question pls?
Are you using "custom" engine?