Incomplete Response Body When Using Envoy to Stream Model Outputs

[XiaozanZhang]

When calling the model with a stream response, an error occurs indicating that the received content is not a complete chunk: (user aibrix project, print envoy rsponse)

`E0603 13:40:50.594592 7 gateway_rsp_body.go:100] "error to unmarshal response" err="unexpected end of JSON input" requestID="a701fd4c-0ac2-4e67-b467-bceae3f3e395" responseBody=<
easoning_content":null,"tool_calls":[{"id":null,"index":0,"type":"function","function":{"name":null,"arguments":"Info" style=""}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"dfe89833d194494f80ec57b762330e28","object":"chat.completion.chunk","created":1780494050,"model":"GLM-5.1-NVFP4","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":[{"id":null,"index":0,"type":"function","function":{"name":null,"arguments":"margin-top:"}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"dfe89833d194494f80ec57b762330e28","object":"chat.completion.chunk","created":1780494050,"model":"GLM-5.1-NVFP4","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":[{"id":null,"index":0,"type":"function","function":{"name":null,"arguments":" 12px;\\\"></dl"}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}`

Steps to Reproduce

envoy v1.33.2
aibrix 0.6.0

[nezdolik]

@XiaozanZhang are you using Envoy AI Gateway? If yes, then please raise this issue in Envoy gateway repo.

[XiaozanZhang]

@nezdolik Hello, I'm using the open source project aibirx(https://github.com/vllm-project/aibrix/releases/tag/v0.6.0), which uses envoy (image: envoyproxy/envoy:v1.33.2) and gateway (image: envoyproxy/gateway:v1.2.8)

[XiaozanZhang]

By capturing packets at the aibirx and model layers, it is found that the content returned by the envoy to gateway-plugins is:

data: {"id":"chatcmpl-77c3ac34-85fa-4b8c-a289-94c51399534e","object":"chat.completion.chunk","created":1780649603,"model":"Qwen2.5-VL-7B-VLLM-test","choices":[{"index":0,"delta":{"content":"Cal","reasoning_content":null},"lB...^..........Y*.. ..ogprobs":null,"finish_reason":null,"token_ids":null}]}

Because these two pieces of data are not JSON, an error is reported.

[KBaichoo]

Hey @XiaozanZhang , they were a variety of fixes for Envoy with ext_proc in more recent versions e.g. issue #41654. Could you bump the envoy to a more recent envoy version to see if the resolves your issue.