Skip to content

KSereve c++ samples with resnet #1439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 39 commits into from
Dec 16, 2022
Merged

KSereve c++ samples with resnet #1439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
95c8eaf
added cmake lists
bstrzele Sep 20, 2022
b651ce0
added infer example with dummy
bstrzele Sep 20, 2022
1fed9f9
args fix
bstrzele Sep 20, 2022
4865ea9
removed redundant defaults
bstrzele Sep 20, 2022
25be8b0
added documentation for grpc infer dummy
bstrzele Sep 23, 2022
7fa9d7f
Apply suggestions from code review
bstrzele Sep 27, 2022
029ecbb
converted ns to ms
bstrzele Sep 28, 2022
34b2672
removed redundant output
bstrzele Sep 28, 2022
6f50b73
added server live to readme
bstrzele Sep 28, 2022
f1c41dc
added mentions of dummy model
bstrzele Sep 30, 2022
ffa64f1
changed file extensions from .cc to .cpp
bstrzele Oct 6, 2022
befbdfc
image classification
bstrzele Sep 27, 2022
8c4b099
converted ns to ms
bstrzele Sep 28, 2022
2e20d13
minor fixes
bstrzele Sep 28, 2022
42e5081
minor fix
bstrzele Sep 28, 2022
5cd182c
added resnet download
bstrzele Sep 30, 2022
5b394bf
changed file extensions from .cc to .cpp
bstrzele Oct 6, 2022
512d9cd
sending native request
bstrzele Oct 26, 2022
3f37cd1
cleanup
bstrzele Dec 2, 2022
d82d8cd
doc fix
bstrzele Dec 2, 2022
237ebef
param fix
bstrzele Dec 2, 2022
1227feb
removed opencv
bstrzele Dec 2, 2022
63252c1
rebase fix
bstrzele Dec 7, 2022
f43945c
added rest infer
bstrzele Dec 7, 2022
964463f
minor doc fix
bstrzele Dec 7, 2022
79019b0
minor fix
bstrzele Dec 7, 2022
9c24add
rest doc fix
bstrzele Dec 7, 2022
5846663
fixed stats
bstrzele Dec 7, 2022
a37426a
doc fix
bstrzele Dec 7, 2022
c3cec7e
rps fix
bstrzele Dec 7, 2022
8ea29ef
updates for documentation test
ngrozae Dec 7, 2022
34ded7d
fixed help msg
bstrzele Dec 7, 2022
3123e34
review fix
bstrzele Dec 14, 2022
f7b75e7
added error handling
bstrzele Dec 15, 2022
90caac6
Update client/cpp/kserve-api/README.md
bstrzele Dec 16, 2022
176993b
fix
bstrzele Dec 16, 2022
f845142
fix
bstrzele Dec 16, 2022
bc944c2
help fix
bstrzele Dec 16, 2022
b4e5c91
rest fix
bstrzele Dec 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
207 changes: 156 additions & 51 deletions client/cpp/kserve-api/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,19 @@ OpenVINO Model Server introduced support for [KServe API](https://github.com/kse

This guide shows how to interact with KServe API endpoints on both gRPC and HTTP interfaces using [Triton](https://github.com/triton-inference-server)'s client library. It covers following topics:
- <a href="#grpc-api">GRPC API Examples </a>
- <a href="#grpc-server-live">grpc_server_live.py</a>
- <a href="#grpc-server-ready">grpc_server_ready.py</a>
- <a href="#grpc-server-metadata">grpc_server_metadata.py</a>
- <a href="#grpc-model-ready">grpc_model_ready.py</a>
- <a href="#grpc-model-metadata">grpc_model_metadata.py</a>
- <a href="#grpc-model-infer">grpc_infer_resnet.py</a>
- <a href="#grpc-model-infer-binary">grpc_infer_binary_resnet.py</a>
- <a href="#grpc-server-live">grpc_server_live</a>
- <a href="#grpc-server-ready">grpc_server_ready</a>
- <a href="#grpc-server-metadata">grpc_server_metadata</a>
- <a href="#grpc-model-ready">grpc_model_ready</a>
- <a href="#grpc-model-metadata">grpc_model_metadata</a>
- <a href="#grpc-model-infer">grpc_infer_dummy</a>
- <a href="#http-api">HTTP API Example</a>
- <a href="#http-server-live">http_server_live.py</a>
- <a href="#http-server-ready">http_server_ready.py</a>
- <a href="#http-server-metadata">http_server_metadata.py</a>
- <a href="#http-model-ready">http_model_ready.py</a>
- <a href="#http-model-metadata">http_model_metadata.py</a>
- <a href="#http-model-infer">http_infer_resnet.py</a>
- <a href="#http-model-infer-binary">http_infer_binary_resnet.py</a>
- <a href="#http-server-live">http_server_live</a>
- <a href="#http-server-ready">http_server_ready</a>
- <a href="#http-server-metadata">http_server_metadata</a>
- <a href="#http-model-ready">http_model_ready</a>
- <a href="#http-model-metadata">http_model_metadata</a>
- <a href="#http-model-infer">http_infer_dummy</a>

## Before you run the samples

Expand All @@ -30,7 +28,7 @@ cd model_server

### Start the Model Server Container with Dummy Model
```Bash
docker run --rm -d -v $(pwd)/src/test/dummy:/models -p 9000:9000 openvino/model_server:latest --model_name dummy --model_path /models --port 9000
docker run --rm -d -v $(pwd)/src/test/dummy:/models -p 9000:9000 -p 8000:8000 openvino/model_server:latest --model_name dummy --model_path /models --port 9000 --rest_port 8000
```

### Build client library and samples
Expand All @@ -47,9 +45,6 @@ cd samples

This section demonstrates inference on a simple model, which increments each provided value.


Once you finish above steps, you are ready to run the samples.

### Run the Client to get server liveness <a name="grpc-server-live"></a>

- Command
Expand Down Expand Up @@ -125,8 +120,8 @@ Usage:

```Bash
./grpc_server_metadata --grpc_port 9000 --grpc_address localhost
Name: "OpenVINO Model Server"
Version: "2022.2.c290da85"
Name: OpenVINO Model Server
Version: 2022.2.c290da85
```

### Run the Client to get model readiness <a name="grpc-model-ready"></a>
Expand Down Expand Up @@ -155,7 +150,7 @@ Usage:
- Usage Example

```Bash
./grpc_model_ready --grpc_port 9000 --grpc_address localhost --model_name resnet
./grpc_model_ready --grpc_port 9000 --grpc_address localhost --model_name dummy
Model Ready: True
```

Expand Down Expand Up @@ -185,24 +180,21 @@ Usage:
- Usage Example

```Bash
./grpc_model_metadata --grpc_port 9000 --grpc_address localhost --model_name resnet
model metadata:
name: "resnet"
./grpc_model_metadata --grpc_port 9000 --grpc_address localhost --model_name dummy
name: "dummy"
versions: "1"
platform: "OpenVINO"
inputs {
name: "0"
name: "b"
datatype: "FP32"
shape: 1
shape: 224
shape: 224
shape: 3
shape: 10
}
outputs {
name: "1463"
name: "a"
datatype: "FP32"
shape: 1
shape: 1000
shape: 10
}
```
### Run the Client to perform inference
Expand Down Expand Up @@ -258,15 +250,15 @@ Requests per second: 189.041

```Bash
./http_server_live --help
Sends requests via KServe rest API to check if server is alive.
Sends requests via KServe REST API to check if server is alive.
Usage:
http_server_live [OPTION...]

-h, --help Show this help message and exit
--http_address HTTP_ADDRESS
Specify url to grpc service. (default:
Specify url to REST service. (default:
localhost)
--http_port PORT Specify port to grpc service. (default:
--http_port PORT Specify port to REST service. (default:
8000)
--timeout TIMEOUT Request timeout. (default: 0)
```
Expand All @@ -284,15 +276,15 @@ Server Live: True

```Bash
./http_server_ready --help
Sends requests via KServe rest API to check if server is ready.
Sends requests via KServe REST API to check if server is ready.
Usage:
http_server_ready [OPTION...]

-h, --help Show this help message and exit
--http_address HTTP_ADDRESS
Specify url to grpc service. (default:
Specify url to REST service. (default:
localhost)
--http_port PORT Specify port to grpc service. (default:
--http_port PORT Specify port to REST service. (default:
8000)
--timeout TIMEOUT Request timeout. (default: 0)
```
Expand All @@ -310,15 +302,15 @@ Server Ready: True

```Bash
./http_server_metadata --help
Sends requests via KServe rest API to get server metadata.
Sends requests via KServe REST API to get server metadata.
Usage:
http_server_metadata [OPTION...]

-h, --help Show this help message and exit
--http_address HTTP_ADDRESS
Specify url to grpc service. (default:
Specify url to REST service. (default:
localhost)
--http_port PORT Specify port to grpc service. (default:
--http_port PORT Specify port to REST service. (default:
8000)
--timeout TIMEOUT Request timeout. (default: 0)
```
Expand All @@ -327,7 +319,7 @@ Usage:

```Bash
./http_server_metadata --http_port 8000 --http_address localhost
{'name': 'OpenVINO Model Server', 'version': '2022.2.c290da85'}
{"name":"OpenVINO Model Server","version":"2022.2.c290da85"}
```

### Run the Client to get model readiness <a name="http-model-ready"></a>
Expand All @@ -336,15 +328,15 @@ Usage:

```Bash
./http_model_ready --help
Sends requests via KServe rest API to check if model is ready for inference.
Sends requests via KServe REST API to check if model is ready for inference.
Usage:
http_model_ready [OPTION...]

-h, --help Show this help message and exit
--http_address HTTP_ADDRESS
Specify url to grpc service. (default:
Specify url to REST service. (default:
localhost)
--http_port PORT Specify port to grpc service. (default:
--http_port PORT Specify port to REST service. (default:
8000)
--model_name MODEL_NAME Define model name, must be same as is in
service. (default: dummy)
Expand All @@ -366,15 +358,15 @@ Model Ready: True

```Bash
./http_model_metadata --help
Sends requests via KServe rest API to get model metadata.
Sends requests via KServe REST API to get model metadata.
Usage:
http_ready [OPTION...]

-h, --help Show this help message and exit
--http_address HTTP_ADDRESS
Specify url to grpc service. (default:
Specify url to REST service. (default:
localhost)
--http_port PORT Specify port to grpc service. (default:
--http_port PORT Specify port to REST service. (default:
8000)
--model_name MODEL_NAME Define model name, must be same as is in
service. (default: dummy)
Expand All @@ -394,16 +386,16 @@ Usage:
- Command

```Bash
./http_infer_resnet --help
Sends requests via KServe rest API.
./http_infer_dummy --help
Sends requests via KServe REST API.
Usage:
http_infer_dummy [OPTION...]

-h, --help Show this help message and exit
--http_address HTTP_ADDRESS
Specify url to grpc service. (default:
Specify url to REST service. (default:
localhost)
--http_port PORT Specify port to grpc service. (default:
--http_port PORT Specify port to REST service. (default:
8000)
--input_name INPUT_NAME Specify input tensor name. (default: b)
--output_name OUTPUT_NAME
Expand All @@ -418,7 +410,7 @@ Usage:
- Usage Example

```Bash
./http_infer_resnety --http_port 8000
./http_infer_dummy --http_port 8000
0 => 1
1 => 2
2 => 3
Expand All @@ -434,4 +426,117 @@ Number of requests: 1
Total processing time: 2.18683 ms
Latency: 2.18683 ms
Requests per second: 457.283
```

## Examples with Resnet Model

### Download the Pretrained Model
Download the model files and store them in the `models` directory
```Bash
mkdir -p models/resnet/1
curl https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.bin https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.xml -o models/resnet/1/resnet50-binary-0001.bin -o models/resnet/1/resnet50-binary-0001.xml
```

### Start the Model Server Container with Resnet Model
```Bash
docker run --rm -d -v $(pwd)/models:/models -p 9000:9000 -p 8000:8000 openvino/model_server:latest --model_name resnet --model_path /models/resnet --port 9000 --rest_port 8000 --layout NHWC:NCHW --plugin_config '{"PERFORMANCE_HINT":"LATENCY"}'
```

Once you finish above steps, you are ready to run the samples.

### Run the Client to perform inference using gRPC API
```Bash
./grpc_infer_resnet --help
Sends requests via KServe gRPC API.
Usage:
grpc_infer_resnet [OPTION...]

-h, --help Show this help message and exit
--images_list IMAGES Path to a file with a list of labeled
images.
--labels_list LABELS Path to a file with a list of labels.
--grpc_address GRPC_ADDRESS
Specify url to grpc service. (default:
localhost)
--grpc_port PORT Specify port to grpc service. (default:
9000)
--input_name INPUT_NAME Specify input tensor name. (default: 0)
--output_name OUTPUT_NAME
Specify input tensor name. (default: 1463)
--model_name MODEL_NAME Define model name, must be same as is in
service. (default: resnet)
--model_version MODEL_VERSION
Define model version.
--timeout TIMEOUT Request timeout. (default: 0)
```

- Usage Example

```Bash
./grpc_infer_resnet --images_list resnet_input_images.txt --labels_list resnet_labels.txt --grpc_port 9000
../../../../demos/common/static/images/airliner.jpeg classified as 404 airliner
../../../../demos/common/static/images/arctic-fox.jpeg classified as 279 Arctic fox, white fox, Alopex lagopus
../../../../demos/common/static/images/bee.jpeg classified as 309 bee
../../../../demos/common/static/images/golden_retriever.jpeg classified as 207 golden retriever
../../../../demos/common/static/images/gorilla.jpeg classified as 366 gorilla, Gorilla gorilla
../../../../demos/common/static/images/magnetic_compass.jpeg classified as 635 magnetic compass
../../../../demos/common/static/images/peacock.jpeg classified as 84 peacock
../../../../demos/common/static/images/pelican.jpeg classified as 144 pelican
../../../../demos/common/static/images/snail.jpeg classified as 113 snail
../../../../demos/common/static/images/zebra.jpeg classified as 340 zebra
Accuracy 100%
======Client Statistics======
Number of requests: 10
Total processing time: 96.651 ms
Latency: 9.6651 ms
Requests per second: 103.465
```

### Run the Client to perform inference using REST API
```Bash
./http_infer_resnet --help
Sends requests via KServe REST API.
Usage:
http_infer_resnet [OPTION...]

-h, --help Show this help message and exit
--images_list IMAGES Path to a file with a list of labeled
images.
--labels_list LABELS Path to a file with a list of labels.
--http_address GRPC_ADDRESS
Specify url to REST service. (default:
localhost)
--http_port PORT Specify port to REST service. (default:
9000)
--input_name INPUT_NAME Specify input tensor name. (default: 0)
--output_name OUTPUT_NAME
Specify input tensor name. (default: 1463)
--model_name MODEL_NAME Define model name, must be same as is in
service. (default: resnet)
--model_version MODEL_VERSION
Define model version.
--timeout TIMEOUT Request timeout. (default: 0)

```

- Usage Example

```Bash
./http_infer_resnet --images_list resnet_input_images.txt --labels_list resnet_labels.txt --http_port 8000
../../../../demos/common/static/images/airliner.jpeg classified as 404 airliner
../../../../demos/common/static/images/zebra.jpeg classified as 340 zebra
../../../../demos/common/static/images/arctic-fox.jpeg classified as 279 Arctic fox, white fox, Alopex lagopus
../../../../demos/common/static/images/bee.jpeg classified as 309 bee
../../../../demos/common/static/images/golden_retriever.jpeg classified as 207 golden retriever
../../../../demos/common/static/images/gorilla.jpeg classified as 366 gorilla, Gorilla gorilla
../../../../demos/common/static/images/magnetic_compass.jpeg classified as 635 magnetic compass
../../../../demos/common/static/images/peacock.jpeg classified as 84 peacock
../../../../demos/common/static/images/pelican.jpeg classified as 144 pelican
../../../../demos/common/static/images/snail.jpeg classified as 113 snail
Accuracy 100%
======Client Statistics======
Number of requests: 10
Total processing time: 115.804 ms
Latency: 11.5804 ms
Requests per second: 86.3526
```
3 changes: 2 additions & 1 deletion client/cpp/kserve-api/samples/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,9 @@ function(sample sample_name)
endfunction()

sample(infer_dummy)
sample(infer_resnet)
sample(server_live)
sample(server_ready)
sample(server_metadata)
sample(model_ready)
sample(model_metadata)
sample(model_metadata)
Loading