Develop and optimize CV/DL applications with Intel OpenVINO toolkit
Download as PPTX, PDF2 likes538 views
OpenVINO is a set of tools and libraries designed for high-performance computer vision and deep learning application development, optimized for Intel platforms. It offers efficient model inference on various hardware, significantly reducing execution footprint and dependencies, while supporting heterogenous execution and customization. The toolkit includes a model zoo with pre-trained models for popular computer vision use cases, alongside extensive samples demonstrating its capabilities.
Develop and optimize CV/DL applications with Intel OpenVINO toolkit
- 2. Intel ConfidentialIntel Confidential Brief OpenVINO™ Introduction • OpenVINO ™ is • set of tools and libraries for CV/DL application developers • high performance, low footprint solution for deployment • API for unified access to CV/DL capabilities of Intel platforms • OpenVINO ™ is not • tool for data scientists • solution for training of deep learning models
- 3. Intel ConfidentialIntel Confidential 3 OpenVINO™ Benefits • Powerful combination of highly optimized Classical CV and DL primitives • Allows to run inference on Intel CPU, GPU, VPU and FPGA • Best performing solution on all Intel architectures • ~2x faster than fastest TensorFlow and MxNet on CPU • Smallest execution footprint (lowest memory consumption) • 2x smaller than MXNet, 4x smaller than TensorFlow • Minimum # of dependencies (no dependencies on training frameworks) • Multiple OS support (Linux and Windows)
- 4. Intel ConfidentialIntel Confidential 4 OpenVINO™ Specifics • Highly optimized implementations of DL primitives • Most efficient on each Intel platform • Focus on inference only • Aggressive layer fusion at the inference step • Including HW accelerated steps tuned for inference • Efficient activation memory reuse • Often close to bare minimum
- 5. Intel ConfidentialIntel Confidential 5 DL Workflow Caffe MXNet TensorFlow Caffe2 PyTorch Serialized trained DL model ONNX MKLDNN Plugin clDNN Plugin FPGA Plugin Myriad Plugin Inference Engine Deploy Application Model Optimizer IR .xml .bin Step 1: Import model from Framework format to Framework independent representation Step 2: Update application to use Inference Engine API Eliminate unnecessary layers, lossless fusion where possible Remove framework dependency Accuracy against original model ensured
- 6. Intel ConfidentialIntel Confidential 6 Customization Capabilities • OpenVINO™ provides good coverage of DL primitives out of the box • Constantly growing list of primitives to support new DL topologies • Frequent releases, substantial additions • Not a problem if something is missing! • Good extension mechanism for adding new primitives • Possible to add proprietary layers, more optimized layers, etc. • Both in Model Optimizer and Inference Engine (import and run)
- 7. Intel ConfidentialIntel Confidential 7 Application Design Workflow Desig n Verify logic Debug Fix Best done on CPU: - Easier to verify - Simpler debugging procedures CHANGE TARGET Check scalability Fix pipeline System testing Best done on Actual target (e.g. VPU): - Exact performance - Correct timings Accuracy and functionality across targets
- 8. Intel ConfidentialIntel Confidential 8 Heterogeneous Execution • When a certain primitive is not supported on a target • Custom proprietary primitive or inefficient HW for a task • Heterogeneous execution ensures full topology execution • Automatic data transfer between targets whenever needed • Work splitting and scheduling • No need to do any manual network manipulations! FPGA CPU
- 9. Intel ConfidentialIntel Confidential 9 Power of Parallel Execution Asynchronous API provides capabilities for: • Running main thread in parallel with ongoing inference (CPU/GPU) • Hiding data transfer latency for accelerators (VPU, FPGA) • Filling accelerators with work Transfer 1 Inference 1 Transfer 2 Transfer 1 Transfer 2 Inference 2 Inference 1 Inference 2 Result 1 Result 1 Result 2 Result 2 Sync API Async API
- 10. Intel ConfidentialIntel Confidential 10 Efficient Frame Preprocessing DECODE Layout Transform Resize to network Detection network Resize to network Object Analysis Network Crop Preprocessing DL Inference • Preprocessing is typically done via manual coding or using libraries • OpenCV is most popular • Fastest OpenCV build is available in OpenVINO™ • Deep Learning Inference Engine encapsulates basic preprocessing capability • Automatic frame resize based on input frame and network size • Automatic layout conversion and cropping • In a nutshell –> Just provide frame and it will be suitable for inference automatically
- 11. Intel ConfidentialIntel Confidential 11 INT8 Quantization for CPU • INT8 provides additional acceleration using AVX-512 • Not all CPU targets support it • Very minor quality/accuracy loss • No retraining is required • No code update is needed IR Dataset Analysis tool Updated IR
- 12. Intel ConfidentialIntel Confidential 12 OpenVINO™ Model Zoo • OpenVINO provides pre-trained DL models for deployment • Lightweight, low compute, real time on Intel platforms • Cover popular CV use cases • Face analysis, Security use-cases (person, vehicle, bicycle detection) • Transportation analytics (road segmentation, vehicle/pedestrian detection)
- 13. Intel ConfidentialIntel Confidential 13 Extensive Set of Samples • Actual examples of applications, not just API demonstration • Switching between targets, work distribution • Multiple models in pipeline with preprocessing • Open Models Zoo demonstration
- 14. 14