基于C++快速使用TensorRT来部署模型项目源码_c++tensorrt,tensorrt源码编译资源-CSDN下载

共232个文件

cpp：54个

py：53个

h：35个

版权申诉

TensorRT

项目源码

5星 · 超过95%的资源 97 浏览量 2022-03-24 23:12:24 上传评论 3 收藏 697KB RAR 举报

在IT领域，尤其是在深度学习应用中，模型的高效部署至关重要。TensorRT是一个由NVIDIA开发的高性能深度学习推理（Inference）优化器和运行时，它能够为深度学习模型提供低延迟、高吞吐量的预测服务。本项目源码以C++为基础，教你如何快速利用TensorRT来部署模型，实现高效的计算性能。 TensorRT的核心功能是构建一个高效的执行图，通过解析ONNX（Open Neural Network Exchange）或TensorFlow、PyTorch等框架的模型，进行模型优化，包括层融合、动态量化等技术，以减少内存消耗和提高运算速度。C++ API提供了对这些功能的直接访问，使得开发者可以灵活地将TensorRT集成到自己的应用程序中。项目源码中可能包含以下几个关键部分： 1. **模型加载**：使用TensorRT的`IGraphBuilder`接口构建网络模型，加载预训练的模型权重。通常，这需要将模型定义（如.pb或.onnx文件）转换为TensorRT可理解的格式。 2. **构建网络**：通过`addInput()`和`addOutput()`方法定义输入和输出，然后逐层添加网络结构。例如，卷积层（`addConvolution()`)、池化层（`addPooling()`)、全连接层（`addFullyConnected()`）等。 3. **配置构建策略**：设置优化级别、内存限制、精度模式（如FP32、FP16或INT8）等，通过`ICudaEngineBuilder`的`setConfig()`方法实现。 4. **引擎创建与序列化**：使用`buildCudaEngine()`方法构建引擎，然后可以序列化到文件，以便后续快速加载。序列化可以显著减少模型加载时间，尤其在嵌入式设备上。 5. **执行上下文与推理**：创建`IExecutionContext`，它是运行推理的核心。设置输入和输出缓冲区，调用`execute()`方法执行推理。 6. **输入和输出处理**：C++代码需要处理输入数据的预处理和输出结果的后处理。例如，归一化、反归一化、维度调整等。 7. **错误处理**：使用TensorRT的异常处理机制来捕获和处理可能出现的错误，确保程序的健壮性。在实际项目中，可能还需要考虑多线程并行执行、GPU资源管理、性能监控等功能。通过分析这个项目源码，你可以深入理解TensorRT的工作原理，以及如何将其高效地应用于C++项目中。基于C++快速使用TensorRT部署模型项目源码是一个实践性很强的学习资源，它涵盖了从模型导入、优化、执行到结果处理的全过程。掌握这些技能，不仅可以提升深度学习模型的部署效率，也能为其他高性能计算任务打下坚实的基础。

资源推荐

资源详情

资源评论

收起资源包目录

基于C++快速使用TensorRT来部署模型项目源码（232个子文件）

setup.cfg 312B

pybind11Tools.cmake 9KB

FindPythonLibsNew.cmake 8KB

CUDA_utils.cmake 4KB

FindEigen3.cmake 3KB

FindCatch.cmake 2KB

test_methods_and_attributes.cpp 23KB

test_virtual_functions.cpp 17KB

test_class.cpp 17KB

test_eigen.cpp 16KB

test_smart_ptr.cpp 16KB

test_numpy_dtypes.cpp 16KB

test_factory_constructors.cpp 16KB

test_numpy_array.cpp 15KB

Trt.cpp 15KB

test_sequences_and_iterators.cpp 13KB

cnpy.cpp 11KB

test_stl.cpp 11KB

test_interpreter.cpp 10KB

test_pytypes.cpp 9KB

test_copy_move.cpp 9KB

test_builtin_casters.cpp 9KB

test_multiple_inheritance.cpp 9KB

tinyexec.cpp 8KB

test_exceptions.cpp 7KB

test_operator_overloading.cpp 6KB

test_callbacks.cpp 6KB

test_buffers.cpp 6KB

CuteSampleDynamicExtPlugin.cpp 6KB

CuteSamplePlugin.cpp 5KB

pybind11_cross_module_tests.cpp 5KB

Int8Calibrator.cpp 5KB

test_pickling.cpp 5KB

test_local_bindings.cpp 4KB

test_tagbased_polymorphic.cpp 4KB

test_kwargs_and_defaults.cpp 4KB

test_constants_and_functions.cpp 4KB

pybind11_tests.cpp 4KB

test_numpy_vectorize.cpp 4KB

test_call_policies.cpp 4KB

test_stl_binders.cpp 3KB

test_modules.cpp 3KB

test_opaque_types.cpp 3KB

sampleDynamicShape.cpp 2KB

test_enum.cpp 2KB

test_eval.cpp 2KB

test_docstring_options.cpp 2KB

plugin_utils.cpp 2KB

test_iostream.cpp 2KB

test_chrono.cpp 2KB

PyTrt.cpp 2KB

cross_module_gil_utils.cpp 2KB

test_gil_scoped.cpp 2KB

test_async.cpp 863B

embed.cpp 654B

sampleINT8.cpp 645B

catch.cpp 637B

test_union.cpp 603B

external_module.cpp 554B

main.cpp 152B

theme_overrides.css 254B

Doxyfile 564B

.gitignore 367B

.gitignore 35B

.gitignore 30B

.gitmodules 202B

.gitmodules 87B

pybind11.h 94KB

cast.h 87KB

numpy.h 66KB

pytypes.h 57KB

common.h 36KB

eigen.h 28KB

class.h 24KB

stl_bind.h 23KB

attr.h 19KB

init.h 16KB

internals.h 14KB

stl.h 14KB

constructor_stats.h 11KB

cnpy.h 11KB

operators.h 9KB

Trt.h 8KB

embed.h 8KB

chrono.h 8KB

iostream.h 6KB

object.h 5KB

buffer_info.h 4KB

eval.h 4KB

functional.h 4KB

descr.h 3KB

CuteSampleDynamicExtPlugin.h 3KB

CuteSamplePlugin.h 3KB

plugin_utils.h 3KB

pybind11_tests.h 2KB

local_bindings.h 2KB

options.h 2KB

complex.h 2KB

utils.h 2KB

typeid.h 1KB

共 232 条

## A Cute custom onnx plugin template Want to implement a custom plugin for your onnx model and don't know where to start? here is a simple template, let's go! ### Step 1 copy this whole CuteSamplePlugin directory to your project's plugin directory, or, as I recommended, implement it with tiny-tensorrt, then port to your own project. ### Step 2 Now check the name of your node that need to be registered as a plugin, you can find its name via [Netron](https://netron.app/), after import your model, you should see your node name here ![image](https://user-images.githubusercontent.com/38289304/104086586-41f2ca00-5294-11eb-8bb6-af2f127908b2.png) now I get the Plugin name(which is the type attribute of the node) is HSigmoid, open CuteSamplePlugin.cpp, find ```c++ static const char* CUTE_PLUGIN_VERSION{"1"}; static const char* CUTE_PLUGIN_NAME{"CuteSamplePlugin"}; ``` edit the CUTE_PLUGIN_NAME to ```c++ static const char* CUTE_PLUGIN_NAME{"HSigmoid"}; ``` ### Step 3 Now if you run it with your own plugin that contains the custom plugin node, it should find your custom plugin(take a look at test/test.cpp). you should get a running output like this ``` root@f44d36162a5e:/tiny-tensorrt/build# ./unit_test --onnx_path ../demo.onnx (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][CuteSamplePluginCreator][Line 155] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getPluginName][Line 162] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getPluginName][Line 162] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getPluginVersion][Line 168] wow I run to here now [2021-01-09 07:50:56.784] [info] create plugin factory [2021-01-09 07:50:56.784] [info] yolo3 params: class: 1, netSize: 416 [2021-01-09 07:50:56.784] [info] upsample params: scale: 2 [2021-01-09 07:50:56.827] [info] build onnx engine from ../demo.onnx... ---------------------------------------------------------------- Input filename: ../demo.onnx ONNX IR version: 0.0.4 Opset version: 9 Producer name: pytorch Producer version: 1.3 Domain: Model version: 0 Doc string: ---------------------------------------------------------------- /workspace/TensorRT/parsers/onnx/ModelImporter.cpp:139: No importer registered for op: HSigmoid. Attempting to import as plugin. /workspace/TensorRT/parsers/onnx/builtin_op_importers.cpp:3716: Searching for plugin: HSigmoid, plugin_version: 1, plugin_namespace: (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getPluginVersion][Line 168] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getFieldNames][Line 174] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][createPlugin][Line 180] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][CuteSamplePlugin][Line 32] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][setPluginNamespace][Line 143] wow I run to here now /workspace/TensorRT/parsers/onnx/builtin_op_importers.cpp:3733: Successfully created plugin: HSigmoid (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getNbOutputs][Line 43] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getOutputDataType][Line 67] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][CuteSamplePlugin][Line 32] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][setPluginNamespace][Line 143] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getNbOutputs][Line 43] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getOutputDataType][Line 67] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][isOutputBroadcastAcrossBatch][Line 94] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getOutputDimensions][Line 49] wow I run to here now -------> (Unnamed Layer* 0) [Constant]_output x (Unnamed Layer* 0) [Constant]_output -------> 2 [2021-01-09 07:50:56.828] [info] fp16 support: false [2021-01-09 07:50:56.828] [info] int8 support: false [2021-01-09 07:50:56.828] [info] Max batchsize: 4 [2021-01-09 07:50:56.828] [info] Max workspace size: 10485760 [2021-01-09 07:50:56.828] [info] Number of DLA core: 0 [2021-01-09 07:50:56.828] [info] Max DLA batchsize: 268435456 [2021-01-09 07:50:56.828] [info] Current use DLA core: 0 [2021-01-09 07:50:56.828] [info] build engine... (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][CuteSamplePlugin][Line 32] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][setPluginNamespace][Line 143] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getPluginType][Line 119] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getPluginType][Line 119] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getPluginType][Line 119] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][supportsFormat][Line 113] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][supportsFormat][Line 113] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][supportsFormat][Line 113] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][supportsFormat][Line 113] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][supportsFormat][Line 113] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getOutputDataType][Line 67] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getOutputDataType][Line 67] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getOutputDataType][Line 67] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][getOutputDataType][Line 67] wow I run to here now Detected 1 inputs and 1 output network tensors. (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp][CuteSamplePlugin][Line 32] wow I run to here now (๑¯◡¯๑) CUSTOM PLUGIN TRACE----> call [/tiny-tensorrt/plugin/CuteSamplePlugin/CuteSamplePlugin.cpp]

评论收藏

内容反馈

版权申诉