Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone! I Am Using Clearml-Serving When I Am Trying To Add New Endpoint Like This

Hi everyone!

I am using clearml-serving

When I am trying to add new endpoint like this

clearml-serving --id <> model add --engine triton --endpoint conformer_encoder --model-id <> --preprocess 'preprocess_encoder.py' --aux-config "./config.pbtxt"

With file 'config.pbtxt' like this:

name: "conformer_encoder"
platform: "onnxruntime_onnx"
default_model_filename: "model.bin"
max_batch_size: 512
input: [
        {
            name: "length"
            data_type: TYPE_INT64
            reshape: {
                shape: []
            }
            dims: [
                1
            ]
        },
        {
            name: "audio_signal"
            data_type: TYPE_FP16
            dims: [
                80,
                -1
            ]
        }
    ]
    output: [
        {
            name: "encoded_lengths"
            data_type: TYPE_INT32
            reshape: {
                shape: []
            }
            dims: [
                1
            ]
        },
        {
            name: "output"
            data_type: TYPE_FP16
            dims: [
                640,
                -1
            ]
        }
]

I get an error in triton container:

[libprotobuf ERROR /tmp/tritonbuild/tritonserver/build/_deps/repo-third-party-build/grpc-repo/src/grpc/third_party/protobuf/src/google/protobuf/text_format.cc:317] Error parsing text-format inference.ModelConfig: 47:9: Non-repeated field "platform" is specified multiple times.
E0905 11:16:22.089991 52 
 Poll failed for model directory 'conformer_encoder': failed to read text proto from /models/conformer_encoder/config.pbtxt

Can somebody explain me what's wrong?

  
  
Posted 7 months ago
Votes Newest

Answers 17


Hi @<1523701205467926528:profile|AgitatedDove14>

My preprocess file:

from typing import Any, Union, Optional, Callable

class Preprocess(object):
    def init(self):
        pass

    def preprocess(
            self,
            body: Union[bytes, dict],
            state: dict, 
            collect_custom_statistics_fn: Optional[Callable[[dict], None]]
        ) -> Any:
        return body["length"], body["audio_signal"]

    def postprocess(
            self,
            data: Any,
            state: dict, 
            collect_custom_statistics_fn: Optional[Callable[[dict], None]]
        ) -> dict:
        return {
            "encoded_lengths": data["encoded_lengths"].tolist(),
            "output": data["output"].tolist()
        }

My request code that returns error:

import numpy as np
import requests
batch = 4
length = 3
body={
    "length": [length] * batch,
    "audio_signal": np.random.randn(batch, 80, length).astype(np.float16).tolist()
}
response = 
(f"<>", json=body)
response.json()
  
  
Posted 7 months ago

@<1569496075083976704:profile|SweetShells3> remove these from your pbtext:

name: "conformer_encoder"
platform: "onnxruntime_onnx"
default_model_filename: "model.bin"

Second, what do you have in your preprocess_encoder.py ?
And where are you getting the Error? (is it from the triton container? or from the Rest request?

  
  
Posted 7 months ago

Hi @<1569496075083976704:profile|SweetShells3> , can you make the basic example work in your setup?

  
  
Posted 7 months ago

@<1523701205467926528:profile|AgitatedDove14> this error appears before postprocess part.

Today I redeployed existing entrypoint with --aux-config "./config.pbtxt" and get the same error

Before:

!clearml-serving --id "<>" model add --engine triton --endpoint 'conformer_joint' --model-id '<>' --preprocess 'preprocess_joint.py' --input-size '[1, 640]' '[640, 1]' --input-name 'encoder_outputs' 'decoder_outputs' --input-type float32 float32 --output-size '[129]' --output-name 'outputs' --output-type float32 --aux-config name=\"conformer_joint\" platform=\"onnxruntime_onnx\" default_model_filename=\"model.bin\" max_batch_size=16 dynamic_batching.max_queue_delay_microseconds=100

After:

!clearml-serving --id "<>" model add --engine triton --endpoint 'conformer_joint' --model-id '<>' --preprocess 'preprocess_joint.py' --aux-config "./config.pbtxt"

config.pbtxt:

default_model_filename: "model.bin"
max_batch_size: 16
dynamic_batching {
    max_queue_delay_microseconds: 100
}
input: [
        {
            name: "encoder_outputs"
            data_type: TYPE_FP32
            dims: [
                1,
                640
            ]
        },
        {
            name: "decoder_outputs"
            data_type: TYPE_FP32
            dims: [
                640,
                1
            ]
        }
    ]
    output: [
        {
            name: "outputs"
            data_type: TYPE_FP32
            dims: [
                129
            ]
        }
]

"before" Entrypoint worked as expected , "After" One returns the same error:
{'detail': "Error processing request: object of type 'NoneType' has no len()"}

Something wrong with config.pbtxt

P.S. I tried wihout default_model_filename line, but still get an error

  
  
Posted 7 months ago

@<1523701205467926528:profile|AgitatedDove14> I think there is no chance to pass config.pbtxt as is.

https://github.com/allegroai/clearml-serving/blob/main/clearml_serving/serving/preprocess_service.py#L358C9-L358C81

In this line, function use self.model_endpoint.input_name (and after that input_name , input_type and input_size ), but there are no such attributes (see endpoint config above) in endpoint - they are in auxiliary_cfg string field

If I understand correctly, we can not use config.pbtxt as I used above, and we have to define inputs and outputs like in clearml-serving examples, and use config.pbtxt only for additional parameters like max_batch_size

  
  
Posted 7 months ago

"After" version in logs is the same as config above. There is no "before" version in logs((

Endpoint config from ClearML triton task:

conformer_joint {
  engine_type = "triton"
  serving_url = "conformer_joint"
  model_id = "<>"
  version = ""
  preprocess_artifact = "py_code_conformer_joint"
  auxiliary_cfg = """default_model_filename: "model.bin"
max_batch_size: 16
dynamic_batching {
    max_queue_delay_microseconds: 100
}
input: [
        {
            name: "encoder_outputs"
            data_type: TYPE_FP32
            dims: [
                1,
                640
            ]
        },
        {
            name: "decoder_outputs"
            data_type: TYPE_FP32
            dims: [
                640,
                1
            ]
        }
    ]
    output: [
        {
            name: "outputs"
            data_type: TYPE_FP32
            dims: [
                129
            ]
        }
]
"""
}
  
  
Posted 7 months ago

@<1523701205467926528:profile|AgitatedDove14> config.pbtxt in triton container (inside /models/conformer_joint) - after merge:

default_model_filename: "model.bin"
max_batch_size: 16
dynamic_batching {
    max_queue_delay_microseconds: 100
}
input: [
        {
            name: "encoder_outputs"
            data_type: TYPE_FP32
            dims: [
                1,
                640
            ]
        },
        {
            name: "decoder_outputs"
            data_type: TYPE_FP32
            dims: [
                640,
                1
            ]
        }
    ]
    output: [
        {
            name: "outputs"
            data_type: TYPE_FP32
            dims: [
                129
            ]
        }
]


platform: "onnxruntime_onnx"
  
  
Posted 7 months ago

I am getting this error in request response:

import numpy as np
import requests
body={
    "encoder_outputs": [np.random.randn(1, 640).tolist()],
    "decoder_outputs": [np.random.randn(640, 1).tolist()]
}
response = 
(f"
", json=body)
response.json()

Unfortunately, I see nothing related to this problem in both inference and triton pods /deployments (we use Kubernetes to spin ClearML-serving

  
  
Posted 7 months ago

  • I'm happy tp hear you found a work around
  • Seems like there is something wrong with the way the pbtxt is being merged, but I need some more information

{'detail': "Error processing request: object of type 'NoneType' has no len()"}

Where are you seeing this error?
What are you seeing in the docker-compose log.

  
  
Posted 7 months ago

I see... In the triton pod, when you run it, it should print the combined pbtxt. Can you print both before/after ones? so that we could compare ?

  
  
Posted 7 months ago

Hi @<1523701205467926528:profile|AgitatedDove14>

Are there any questions or updates about the issue?

  
  
Posted 7 months ago

Thanks @<1569496075083976704:profile|SweetShells3> for bumping it!
Let me check where it stands, I think I remember a fix...

  
  
Posted 7 months ago

Thanks @<1569496075083976704:profile|SweetShells3> ! let me see if I can reproduce the issue

  
  
Posted 7 months ago

Yeah I think that for some reason the merge of the pbtxt raw file is not working.
Any chance you have an end to end example we could debug? (maybe just add a pbtxt for one of the examples?)

  
  
Posted 7 months ago

Hi @<1523701205467926528:profile|AgitatedDove14>

https://github.com/allegroai/clearml-serving/issues/62

I have an issue basen on that case. Could you tell me if I miss something in it?

  
  
Posted 7 months ago

 data["encoded_lengths"]

This makes no sense to me, data is a numpy array, not a pandas frame...

  
  
Posted 7 months ago

Hi @<1523701087100473344:profile|SuccessfulKoala55> Turns out if I delete

platform: ... 

string from config.pbtxt, it will deploy model on tritonserver (serving v 1.3.0 add "platform" string at the end of config file when clearm-model has "framework" attribute). But when I try to check endpoint with random data (but with right shape according config), I am getting

{'detail': "Error processing request: object of type 'NoneType' has no len()"}

error. Do you know how to solve it?

  
  
Posted 7 months ago