|
|
5.3.38
IIfConditionalInputLayer
95
5.3.39
IEinsumLayer
95
5.3.40
IAssertionLayer
96
5.3.41
IOneHotLayer
96
5.3.42
INonZeroLayer
96
5.3.43
INMSLayer
96
5.3.44
IReverseSequenceLayer
98
5.3.45
INormalizationLayer
98
6
Plugin
99
6.1
IPluginCreator
99
6.2
IPluginRegistry
101
7
Int8
103
7.1
IInt8Calibrator
103
7.2
IInt8LegacyCalibrator
105
7.3
IInt8EntropyCalibrator
106
7.4
IInt8EntropyCalibrator2
108
7.5
IInt8MinMaxCalibrator
109
8
Algorithm Selector
111
9
UFF Parser
115
9.1
Fields
116
10
Caffe Parser
119
10.1
Plugins
120
11
Onnx Parser
121
12
UFF Converter
125
12.1
Conversion Tools
125
12.1.1
Tensorflow Modelstream to UFF
125
12.1.2
Tensorflow Frozen Protobuf Model to UFF
126
13
UFF Operators
127
13.1
Input
127
13.1.1
Supported Datatypes
127
13.2
Identity
127
13.2.1
Inputs
127
13.2.2
Supported Datatypes
127
13.3
Const
127
13.3.1
Supported Datatypes
128
13.4
Conv
128
13.4.1
Inputs
128
13.4.2
Attributes
128
13.4.3
Supported Datatypes
128
13.5
ConvTranspose
128
13.5.1
Inputs
128
13.5.2
Attributes
129
13.5.3
Supported Datatypes
129
13.6
Pool
129
13.6.1
Inputs
129
13.6.2
Attributes
129
13.6.3
Supported Datatypes
129
iii
13.7
FullyConnected . .
129
13.7.1
Inputs . .
129
13.7.2
Supported Datatypes
130
13.8
LRN
130
13.8.1
Inputs . .
130
13.8.2
Attributes
130
13.8.3
Supported Datatypes
130
13.9
Binary
130
13.9.1
Inputs . .
130
13.9.2
Attributes
130
13.9.3
Supported Datatypes
131
13.10
Unary
131
13.10.1 Inputs . .
131
13.10.2 Attributes
131
13.10.3 Supported Datatypes
131
13.11
Reshape
131
13.11.1 Inputs . .
131
13.11.2 Supported Datatypes
132
13.12
ExpandDims
132
13.12.1 Inputs . .
132
13.12.2 Attributes
132
13.12.3 Supported Datatypes
132
13.13
ArgMax
132
13.13.1 Inputs . .
132
13.13.2 Attributes
132
13.13.3 Supported Datatypes
133
13.14
ArgMin
133
13.14.1 Inputs . .
133
13.14.2 Attributes
133
13.14.3 Supported Datatypes
133
13.15
Transpose
133
13.15.1 Inputs . .
133
13.15.2 Attributes
133
13.15.3 Supported Datatypes
133
13.16
Reduce
134
13.16.1 Inputs . .
134
13.16.2 Attributes
134
13.16.3 Supported Datatypes
134
13.17
Concat
134
13.17.1 Inputs . .
134
13.17.2 Attributes
134
13.17.3 Supported Datatypes
135
13.18
MarkOutput
135
13.18.1 Inputs . .
135
13.18.2 Supported Datatypes
135
13.19
Activation
135
13.19.1 Inputs . .
135
13.19.2 Attributes
135
13.19.3 Supported Datatypes
135
13.20
Softmax
135
13.20.1 Inputs . .
136
13.20.2 Attributes
136
13.20.3 Supported Datatypes
136
13.21
BatchNorm
136
iv
13.21.1 Inputs
136
13.21.2 Attributes
136
13.21.3 Supported Datatypes
136
13.22 Shape
136
13.22.1 Inputs
137
13.22.2 Supported Datatypes
137
13.23 StridedSlice
137
13.23.1 Inputs
137
13.23.2 Attributes
137
13.23.3 Supported Datatypes
137
13.24 Stack
137
13.24.1 Inputs
138
13.24.2 Attributes
138
13.24.3 Supported Datatypes
138
13.25 Squeeze
138
13.26 Flatten
138
13.26.1 Inputs
138
13.26.2 Supported Datatypes
138
13.27 Pad
138
13.27.1 Inputs
138
13.27.2 Supported Datatypes
139
13.28 Gather
139
13.28.1 Inputs
139
13.28.2 Supported Datatypes
139
13.29 GatherV2
139
13.29.1 Inputs
139
13.29.2 Attributes
139
13.29.3 Supported Datatypes
139
14 Graph Surgeon
141
14.1
Node Creation
141
14.2
Static Graph
142
14.3
Dynamic Graph (Inherits from StaticGraph) .
144
Index
147
v
vi
CHAPTER
ONE
GETTING STARTED WITH TENSORRT
1.1 Installation
For installation instructions, please refer to https://docs.nvidia.com/deeplearning/sdk/tensorrt-install-guide/index.html
1.2 Samples
For information about samples, please refer to https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/
index.html#python_samples_section
1.3 Installing PyCUDA
Although not required by the TensorRT Python API, PyCUDA is used in several samples. For installation instructions,
please refer to https://wiki.tiker.net/PyCuda/Installation
1
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
2
Chapter 1. Getting Started with TensorRT
CHAPTER
TWO
CORE CONCEPTS
2.1 TensorRT Workflow
The general TensorRT workflow consists of 3 steps:
1. Populate a tensorrt.INetworkDefinition either with a parser or by using the TensorRT Network API (see
tensorrt.INetworkDefinition for more details). The tensorrt.Builder can be used to generate an
empty tensorrt.INetworkDefinition .
2. Use the tensorrt.Builder to build a tensorrt.ICudaEngine using the populated tensorrt.
INetworkDefinition .
3. Create a tensorrt.IExecutionContext from the tensorrt.ICudaEngine and use it to perform optimized
inference.
2.2 Classes Overview
2.2.1 Logger
Most other TensorRT classes use a logger to report errors, warnings and informative messages. TensorRT provides a
basic tensorrt.Logger implementation, but you can write your own implementation by deriving from tensorrt.
ILogger for more advanced functionality.
2.2.2 Parsers
Parsers are used to populate a tensorrt.INetworkDefinition from a model trained in a Deep Learning framework.
2.2.3 Network
The tensorrt.INetworkDefinition represents a computational graph. In order to populate the network, TensorRT
provides a suite of parsers for a variety of Deep Learning frameworks. It is also possible to populate the network
manually using the Network API.
3
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
2.2.4 Builder
The tensorrt.Builder is used to build a tensorrt.ICudaEngine . In order to do so, it must be provided a popu-
lated tensorrt.INetworkDefinition .
2.2.5 Engine and Context
The tensorrt.ICudaEngine is the output of the TensorRT optimizer. It is used to generate a tensorrt.
IExecutionContext that can perform inference.
4
Chapter 2. Core Concepts
CHAPTER
THREE
FOUNDATIONAL TYPES
3.1 DataType
tensorrt.DataType
Represents data types.
ivar itemsize int The size in bytes of this DataType .
Members:
FLOAT : 32-bit floating point format.
HALF : IEEE 16-bit floating-point format.
INT8 : Signed 8-bit integer representing a quantized floating-point value.
INT32 : Signed 32-bit integer format.
BOOL : 8-bit boolean. 0 = false, 1 = true, other values undefined.
UINT8 : Unsigned 8-bit integer format. Cannot be used to represent quantized floating-point values.
Use the IdentityLayer to convert uint8 network-level inputs to {float32, float16} prior to use
with other TensorRT layers, or to convert intermediate output before uint8 network-level out-
puts from {float32, float16} to uint8. uint8 conversions are only supported for {float32,
float16}. uint8 to {float32, float16} conversion will convert the integer values to equiv-
alent floating point values. {float32, float16} to uint8 conversion will convert the floating
point values to integer values by truncating towards zero. This conversion has undefined behavior
for floating point values outside the range [0.0f, 256.0) after truncation. uint8 conversions are
not supported for {int8, int32, bool}.
FP8 : Signed 8-bit floating point with 1 sign bit, 4 exponent bits, 3 mantissa bits, and exponent-bias
7.
Warning: fp8 is not supported yet and will result in an error or undefined behavior.
TensorRT also exposes some short-hand, NumPy-style DataType aliases that can be used across the library:
Type
Alias
tensorrt.DataType.FLOAT
tensorrt.float32
tensorrt.DataType.HALF
tensorrt.float16
tensorrt.DataType.INT32
tensorrt.int32
tensorrt.DataType.INT8
tensorrt.int8
tensorrt.DataType.BOOL
tensorrt.bool
5
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
tensorrt.nptype(trt_type)
Returns the numpy-equivalent of a TensorRT DataType .
Parameters trt_type - The TensorRT data type to convert.
Returns The equivalent numpy type.
3.2 Weights
tensorrt.WeightsRole
How a layer uses particular Weights. The power weights of an IScaleLayer are omitted. Refitting those is not
supported.
Members:
KERNEL
:
Kernel
for
IConvolutionLayer
,
IDeconvolutionLayer
,
or
IFullyConnectedLayer .
BIAS : Bias for IConvolutionLayer , IDeconvolutionLayer , or IFullyConnectedLayer .
SHIFT : Shift part of IScaleLayer .
SCALE : Scale part of IScaleLayer .
CONSTANT : Weights for IConstantLayer .
ANY : Any other weights role.
class tensorrt.Weights(*args, **kwargs)
An array of weights used as a layer parameter. The weights are held by reference until the engine has been built
- deep copies are not made automatically.
Variables
• dtype - DataType The type of the weights.
• size - int The number of weights in the array.
• nbytes - int Total bytes consumed by the elements of the weights buffer.
Overloaded function.
1.
__init__(self: tensorrt.tensorrt.Weights, type: tensorrt.tensorrt.DataType = <DataType.FLOAT: 0>) ->
None
Initializes an empty (0-length) Weights object with the specified type.
type A type to initialize the weights with. Default: tensorrt.float32
2.
__init__(self: tensorrt.tensorrt.Weights, a: numpy.ndarray) -> None
a A numpy array whose values to use. No deep copies are made.
numpy(self: tensorrt.tensorrt.Weights) → numpy.ndarray
Create a numpy array using the underlying buffer of this weights object.
Returns A new numpy array that holds a reference to this weight object’s buffer - no deep copy
is made.
6
Chapter 3. Foundational Types
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
3.3 Dims
3.3.1 Volume
tensorrt.volume(iterable)
Computes the volume of an iterable.
Parameters iterable - Any python iterable, including a Dims object.
Returns The volume of the iterable. This will return 1 for empty iterables, as a scalar has an empty
shape and the volume of a tensor with empty shape is 1.
3.3.2 Dims
class tensorrt.Dims(*args, **kwargs)
Structure to define the dimensions of a tensor. Dims and all derived classes behave like Python tuple s. Fur-
thermore, the TensorRT API can implicitly convert Python iterables to Dims objects, so tuple or list can be
used in place of this class.
Overloaded function.
1.
__init__(self: tensorrt.tensorrt.Dims) -> None
2.
__init__(self: tensorrt.tensorrt.Dims, shape: List[int]) -> None
property MAX_DIMS
The maximum number of dimensions supported by Dims.
3.3.3 Dims2
class tensorrt.Dims2(*args, **kwargs)
Structure to define 2D shape.
Overloaded function.
1.
__init__(self: tensorrt.tensorrt.Dims2) -> None
2.
__init__(self: tensorrt.tensorrt.Dims2, dim0: int, dim1: int) -> None
3.
__init__(self: tensorrt.tensorrt.Dims2, shape: List[int]) -> None
3.3.4 DimsHW
class tensorrt.DimsHW(*args, **kwargs)
Structure to define 2D shape with height and width.
Variables
• h - int The first dimension (height).
• w - int The second dimension (width).
Overloaded function.
1.
__init__(self: tensorrt.tensorrt.DimsHW) -> None
2.
__init__(self: tensorrt.tensorrt.DimsHW, h: int, w: int) -> None
3.3. Dims
7
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
3.
__init__(self: tensorrt.tensorrt.DimsHW, shape: List[int]) -> None
3.3.5 Dims3
class tensorrt.Dims3(*args, **kwargs)
Structure to define 3D shape.
Overloaded function.
1.
__init__(self: tensorrt.tensorrt.Dims3) -> None
2.
__init__(self: tensorrt.tensorrt.Dims3, dim0: int, dim1: int, dim2: int) -> None
3.
__init__(self: tensorrt.tensorrt.Dims3, shape: List[int]) -> None
3.3.6 Dims4
class tensorrt.Dims4(*args, **kwargs)
Structure to define 4D tensor.
Overloaded function.
1.
__init__(self: tensorrt.tensorrt.Dims4) -> None
2.
__init__(self: tensorrt.tensorrt.Dims4, dim0: int, dim1: int, dim2: int, dim3: int) -> None
3.
__init__(self: tensorrt.tensorrt.Dims4, shape: List[int]) -> None
3.4 IHostMemory
class tensorrt.IHostMemory
Handles library allocated memory that is accessible to the user.
The memory allocated via the host memory object is owned by the library and will be de-allocated when object
is destroyed.
This class exposes a buffer interface using Python’s buffer protocol.
Variables
• dtype - DataType The data type of this buffer.
• nbytes - int Total bytes consumed by the elements of the buffer.
__del__(self: tensorrt.tensorrt.IHostMemory) → None
__exit__(exc_type, exc_value, traceback)
Context managers are deprecated and have no effect. Objects are automatically freed when the reference
count reaches 0.
__init__(*args, **kwargs)
8
Chapter 3. Foundational Types
CHAPTER
FOUR
CORE
4.1 Logger
class tensorrt.ILogger(self: tensorrt.tensorrt.ILogger) → None
Abstract base Logger class for the Builder, ICudaEngine and Runtime .
To implement a custom logger, ensure that you explicitly instantiate the base class in __init__() :
class MyLogger(trt.ILogger):
def __init__(self):
trt.ILogger.__init__(self)
def log(self, severity, msg):
... # Your implementation here
Parameters min_severity - The initial minimum severity of this Logger.
Variables min_severity - Logger.Severity This minimum required severity of messages for
the logger to log them.
The logger used to create an instance of IBuilder, IRuntime or IRefitter is used for all objects created through
that interface. The logger should be valid until all objects created are released.
class Severity(self: tensorrt.tensorrt.ILogger.Severity, value: int) → None
Indicates the severity of a message.
The values in this enum are also accessible in
the ILogger directly. For example, tensorrt.ILogger.INFO corresponds to tensorrt.
ILogger.Severity.INFO .
Members:
INTERNAL_ERROR : Represents an internal error. Execution is unrecoverable.
ERROR : Represents an application error.
WARNING : Represents an application error that TensorRT has recovered from or fallen back
to a default.
INFO : Represents informational messages.
VERBOSE : Verbose messages with debugging information.
property name
log(self: tensorrt.tensorrt.ILogger, severity: nvinfer1::ILogger::Severity, msg: str) → None
Logs a message to stderr . This function must be overriden by a derived class.
9
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Parameters
• severity - The severity of the message.
• msg - The log message.
class tensorrt.Logger(self: tensorrt.tensorrt.Logger, min_severity: tensorrt.tensorrt.ILogger.Severity =
<Severity.WARNING: 2>) → None
Logger for the Builder, ICudaEngine and Runtime .
Parameters min_severity - The initial minimum severity of this Logger.
Variables min_severity - Logger.Severity This minimum required severity of messages for
the logger to log them.
log(self: tensorrt.tensorrt.Logger, severity: tensorrt.tensorrt.ILogger.Severity, msg: str) → None
Logs a message to stderr .
Parameters
• severity - The severity of the message.
• msg - The log message.
4.2 Profiler
class tensorrt.IProfiler(self: tensorrt.tensorrt.IProfiler) → None
Abstract base Profiler class.
To implement a custom profiler, ensure that you explicitly instantiate the base class in __init__() :
class MyProfiler(trt.IProfiler):
def __init__(self):
trt.IProfiler.__init__(self)
def report_layer_time(self, layer_name, ms):
... # Your implementation here
When this class is added to an IExecutionContext, the profiler will be called once per layer for each invocation
of IExecutionContext.execute_v2() or IExecutionContext.execute_async_v2().
It is not recommended to run inference with profiler enabled when the inference execution time is critical since
the profiler may affect execution time negatively.
report_layer_time(self: tensorrt.tensorrt.IProfiler, layer_name: str, ms: float) → None
Reports time in milliseconds for each layer. This function must be overriden a derived class.
Parameters
• layer_name - The name of the layer, set when constructing the INetworkDefinition
. If the engine is built with profiling verbosity set to NONE, the layerName is the decimal
index of the layer.
• ms - The time in milliseconds to execute the layer.
class tensorrt.Profiler(self: tensorrt.tensorrt.Profiler) → None
When this class is added to an IExecutionContext, the profiler will be called once per layer for each invocation
of IExecutionContext.execute_v2() or IExecutionContext.execute_async_v2().
It is not recommended to run inference with profiler enabled when the inference execution time is critical since
the profiler may affect execution time negatively.
10
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
report_layer_time(self: tensorrt.tensorrt.Profiler, layer_name: str, ms: float) → None
Prints time in milliseconds for each layer to stdout.
Parameters
• layer_name - The name of the layer, set when constructing the INetworkDefinition .
• ms - The time in milliseconds to execute the layer.
4.3 IOptimizationProfile
class tensorrt.IOptimizationProfile
Optimization profile for dynamic input dimensions and shape tensors.
When building an ICudaEngine from an INetworkDefinition that has dynamically resizable inputs (at least
one input tensor has one or more of its dimensions specified as -1) or shape input tensors, users need to specify
at least one optimization profile. Optimization profiles are numbered 0, 1, . . .
The first optimization profile that has been defined (with index 0) will be used by the ICudaEngine whenever
no optimization profile has been selected explicitly. If none of the inputs are dynamic, the default optimiza-
tion profile will be generated automatically unless it is explicitly provided by the user (this is possible but not
required in this case). If more than a single optimization profile is defined, users may set a target how much
additional weight space should be maximally allocated to each additional profile (as a fraction of the maximum,
unconstrained memory).
Users set optimum input tensor dimensions, as well as minimum and maximum input tensor dimensions. The
builder selects the kernels that result in the lowest runtime for the optimum input tensor dimensions, and are
valid for all input tensor sizes in the valid range between minimum and maximum dimensions. A runtime error
will be raised if the input tensor dimensions fall outside the valid range for this profile. Likewise, users provide
minimum, optimum, and maximum values for all shape tensor input values.
IOptimizationProfile implements __nonzero__() and __bool__() such that evaluating a profile as a
bool (e.g. if profile:) will check whether the optimization profile can be passed to an IBuilderConfig object.
This will perform partial validation, by e.g. checking that the maximum dimensions are at least as large as the
optimum dimensions, and that the optimum dimensions are always as least as large as the minimum dimensions.
Some validation steps require knowledge of the network definition and are deferred to engine build time.
Variables extra_memory_target - Additional memory that the builder should aim to maximally
allocate for this profile, as a fraction of the memory it would use if the user did not im-
pose any constraints on memory. This unconstrained case is the default; it corresponds to
extra_memory_target == 1.0. If extra_memory_target == 0.0, the builder aims to cre-
ate the new optimization profile without allocating any additional weight memory. Valid inputs
lie between 0.0 and 1.0. This parameter is only a hint, and TensorRT does not guarantee that
the extra_memory_target will be reached. This parameter is ignored for the first (default)
optimization profile that is defined.
get_shape(self: tensorrt.tensorrt.IOptimizationProfile, input: str) → List[tensorrt.tensorrt.Dims]
Get the minimum/optimum/maximum dimensions for a dynamic input tensor. If the dimensions have not
been previously set via set_shape(), return an invalid Dims with a length of -1.
Returns A List[Dims] of length 3, containing the minimum, optimum, and maximum shapes,
in that order. If the shapes have not been set yet, an empty list is returned.
get_shape_input(self: tensorrt.tensorrt.IOptimizationProfile, input: str) → List[List[int]]
Get the minimum/optimum/maximum values for a shape input tensor.
Returns A List[List[int]] of length 3, containing the minimum, optimum, and maximum
values, in that order. If the values have not been set yet, an empty list is returned.
4.3. IOptimizationProfile
11
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
set_shape(self: tensorrt.tensorrt.IOptimizationProfile, input: str, min: tensorrt.tensorrt.Dims, opt:
tensorrt.tensorrt.Dims, max: tensorrt.tensorrt.Dims) → None
Set the minimum/optimum/maximum dimensions for a dynamic input tensor.
This function must be called for any network input tensor that has dynamic dimensions. If min, opt, and
max are the minimum, optimum, and maximum dimensions, and real_shape is the shape for this input
tensor provided to the INetworkDefinition ,then the following conditions must hold:
(1) len(min) == len(opt) == len(max) == len(real_shape)
(2)
0 <= min[i] <= opt[i] <= max[i] for all i
(3) if real_shape[i] != -1, then min[i] == opt[i] == max[i] == real_shape[i]
This function may (but need not be) called for an input tensor that does not have dynamic dimensions. In
this case, all shapes must equal real_shape.
Parameters
• input - The name of the input tensor.
• min - The minimum dimensions for this input tensor.
• opt - The optimum dimensions for this input tensor.
• max - The maximum dimensions for this input tensor.
Raises ValueError if an inconsistency was detected. Note that inputs can be validated only
partially; a full validation is performed at engine build time.
set_shape_input(self: tensorrt.tensorrt.IOptimizationProfile, input: str, min: List[int], opt: List[int], max:
List[int]) → None
Set the minimum/optimum/maximum values for a shape input tensor.
This function must be called for every input tensor t that is a shape tensor (t.is_shape == True). This
implies that the datatype of t is int32, the rank is either 0 or 1, and the dimensions of t are fixed at network
definition time. This function must NOT be called for any input tensor that is not a shape tensor.
If min, opt, and max are the minimum, optimum, and maximum values, it must be true that min[i] <=
opt[i] <= max[i] for all i.
Parameters
• input - The name of the input tensor.
• min - The minimum values for this shape tensor.
• opt - The optimum values for this shape tensor.
• max - The maximum values for this shape tensor.
Raises ValueError if an inconsistency was detected. Note that inputs can be validated only
partially; a full validation is performed at engine build time.
12
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
4.4 IBuilderConfig
tensorrt.QuantizationFlag
List of valid flags for quantizing the network to int8.
Members:
CALIBRATE_BEFORE_FUSION : Run int8 calibration pass before layer fusion. Only valid for
IInt8LegacyCalibrator and IInt8EntropyCalibrator. We always run int8 calibration pass before layer
fusion for IInt8MinMaxCalibrator and IInt8EntropyCalibrator2. Disabled by default.
tensorrt.DeviceType
Device types that TensorRT can execute on
Members:
GPU : GPU device
DLA : DLA core
tensorrt.ProfilingVerbosity
Profiling verbosity in NVTX annotations and the engine inspector
Members:
LAYER_NAMES_ONLY : Print only the layer names. This is the default setting.
DETAILED : Print detailed layer information including layer names and layer parameters.
NONE : Do not print any layer information.
DEFAULT : [DEPRECATED] Same as LAYER_NAMES_ONLY.
VERBOSE : [DEPRECATED] Same as DETAILED.
tensorrt.TacticSource
Tactic sources that can provide tactics for TensorRT.
Members:
CUBLAS : Enables cuBLAS tactics. Enabled by default. NOTE: Disabling this value will cause
the cublas handle passed to plugins in attachToContext to be null.
CUBLAS_LT : Enables cuBLAS LT tactics. Enabled for x86 platforms and only enabled for non-
x86 platforms when CUDA >= 11.0 by default
CUDNN : Enables cuDNN tactics. Enabled by default.
EDGE_MASK_CONVOLUTIONS : Enables convolution tactics implemented with edge mask ta-
bles. These tactics tradeoff memory for performance by consuming additional memory space
proportional to the input size. Enabled by default.
JIT_CONVOLUTIONS : Enables convolution tactics implemented with source-code JIT fusion.
The engine building time may increase when this is enabled. Enabled by default.
tensorrt.EngineCapability
List of supported engine capability flows. The EngineCapability determines the restrictions of a network dur-
ing build time and what runtime it targets. When BuilderFlag::kSAFETY_SCOPE is not set (by default),
EngineCapability.STANDARD does not provide any restrictions on functionality and the resulting seri-
alized engine can be executed with TensorRT’s standard runtime APIs in the nvinfer1 namespace. En-
gineCapability.SAFETY provides a restricted subset of network operations that are safety certified and the
resulting serialized engine can be executed with TensorRT’s safe runtime APIs in the nvinfer1::safe names-
pace. EngineCapability.DLA_STANDALONE provides a restricted subset of network operations that are
4.4. IBuilderConfig
13
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
DLA compatible and the resulting serialized engine can be executed using standalone DLA runtime APIs.
See sampleCudla for an example of integrating cuDLA APIs with TensorRT APIs.
Members:
DEFAULT : [DEPRECATED] Unrestricted: TensorRT mode without any restrictions using TensorRT
nvinfer1 APIs.
SAFE_GPU : [DEPRECATED] Safety-restricted: TensorRT mode for GPU devices using TensorRT
safety APIs. See safety documentation for list of supported layers and formats.
SAFE_DLA : [DEPRECATED] DLA-restricted: TensorRT mode for DLA devices using cuDLA
APIs. Only FP16 and Int8 modes are supported.
STANDARD : Standard: TensorRT flow without targeting the standard runtime. This flow supports
both DeviceType::kGPU and DeviceType::kDLA.
SAFETY : Safety: TensorRT flow with restrictions targeting the safety runtime. See safety documen-
tation for list of supported layers and formats. This flow supports only DeviceType::kGPU.
DLA_STANDALONE : DLA Standalone: TensorRT flow with restrictions targeting external, to Ten-
sorRT, DLA runtimes. See DLA documentation for list of supported layers and formats. This flow
supports only DeviceType::kDLA.
tensorrt.BuilderFlag
Valid modes that the builder can enable when creating an engine from a network definition.
Members:
FP16 : Enable FP16 layer selection
INT8 : Enable Int8 layer selection
DEBUG : Enable debugging of layers via synchronizing after every layer
GPU_FALLBACK : Enable layers marked to execute on GPU if layer cannot execute on DLA
STRICT_TYPES : [DEPRECATED] Enables strict type constraints. Equivalent to setting PRE-
FER_PRECISION_CONSTRAINTS, DIRECT_IO, and REJECT_EMPTY_ALGORITHMS.
REFIT : Enable building a refittable engine
DISABLE_TIMING_CACHE : Disable reuse of timing information across identical layers.
TF32 : Allow (but not require) computations on tensors of type DataType.FLOAT to use TF32. TF32
computes inner products by rounding the inputs to 10-bit mantissas before multiplying, but accumu-
lates the sum using 23-bit mantissas. Enabled by default.
SPARSE_WEIGHTS : Allow the builder to examine weights and use optimized functions when
weights have suitable sparsity.
SAFETY_SCOPE : Change the allowed parameters in the EngineCapability.STANDARD flow to
match the restrictions that EngineCapability.SAFETY check against for DeviceType.GPU and En-
gineCapability.DLA_STANDALONE check against the DeviceType.DLA case. This flag is forced to
true if EngineCapability.SAFETY at build time if it is unset.
OBEY_PRECISION_CONSTRAINTS : Require that layers execute in specified precisions. Build
fails otherwise.
PREFER_PRECISION_CONSTRAINTS : Prefer that layers execute in specified precisions. Fall back
(with warning) to another precision if build would otherwise fail.
DIRECT_IO : Require that no reformats be inserted between a layer and a network I/O tensor for which
ITensor.allowed_formats was set. Build fails if a reformat is required for functional correctness.
14
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
REJECT_EMPTY_ALGORITHMS : Fail if IAlgorithmSelector.select_algorithms returns an empty
set of algorithms.
ENABLE_TACTIC_HEURISTIC : [DEPRECATED] Enable heuristic-based tactic selection for
shorter engine generation time. The performance of the generated engine may not be as performant
as a profiling-based builder.
VERSION_COMPATIBLE : Restrict to lean runtime operators to provide version forward compati-
bility for the plan files.
EXCLUDE_LEAN_RUNTIME : Exclude lean runtime from the plan.
FP8 : Enable FP8 layer selection
tensorrt.PreviewFeature
List of Preview Features that can be enabled. Preview Features have been fully tested but are not yet as stable as other feat
They are provided as opt-in features for at least one release. For example, to enable faster dynamic shapes,
call set_preview_feature() with PreviewFeature.FASTER_DYNAMIC_SHAPES_0805
Members:
FASTER_DYNAMIC_SHAPES_0805 : [DEPRECATED - will be removed in TensorRT 9.0] Op-
timize runtime dimensions with TensorRT’s DL Compiler. Potentially reduces run time and
decreases device memory usage and engine size. Models most likely to benefit from enabling
FASTER_DYNAMIC_SHAPES_0805 are transformer-based models, and models containing dy-
namic control flows. The default value for this flag is on. Turning it off is deprecated.
DISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805 : Disable
usage
of
cuDNN/cuBLAS/cuBLASLt tactics in the TensorRT core library. When the flag is en-
abled, TensorRT core will not use these tactics even if they are specified in set_tactic_sources,
but cudnnContext and cublasContext handles will still be passed to plugins via IPlug-
inV2::attachToContext() if the appropriate tactic sources are set. This allows users to experiment
with disabling external library tactics without having to modify their application’s plugins to
support nullptr handles. The default value for this flag is off.
PROFILE_SHARING_0806 : Allows optimization profiles to be shared across execution contexts.
This will become the default behavior in TensorRT 9.0 and the flag defaults to false.
tensorrt.MemoryPoolType
The type for memory pools used by TensorRT.
Members:
WORKSPACE : WORKSPACE is used by TensorRT to store intermediate buffers within an oper-
ation. This is equivalent to the deprecated IBuilderConfig.max_workspace_size and overrides
that value. This defaults to max device memory. Set to a smaller value to restrict tactics that use
over the threshold en masse. For more targeted removal of tactics use the IAlgorithmSelector
interface.
DLA_MANAGED_SRAM : DLA_MANAGED_SRAM is a fast software managed RAM used by
DLA to communicate within a layer. The size of this pool must be at least 4 KiB and must be a
power of 2. This defaults to 1 MiB. Orin has capacity of 1 MiB per core, and Xavier shares 4
MiB across all of its accelerator cores.
DLA_LOCAL_DRAM : DLA_LOCAL_DRAM is host RAM used by DLA to share intermediate
tensor data across operations. The size of this pool must be at least 4 KiB and must be a power
of 2. This defaults to 1 GiB.
4.4. IBuilderConfig
15
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
DLA_GLOBAL_DRAM : DLA_GLOBAL_DRAM is host RAM used by DLA to store weights
and metadata for execution. The size of this pool must be at least 4 KiB and must be a power of
2. This defaults to 512 MiB.
TACTIC_DRAM : kTACTIC_DRAM is the host DRAM used by the optimizer to run tactics. On
embedded devices, where host and device memory are unified, this includes all device mem-
ory required by TensorRT to build the network up to the point of each memory allocation. This
defaults to 75% of totalGlobalMem as reported by cudaGetDeviceProperties when cudaGetDe-
viceProperties.embedded is true, and 100% otherwise.
tensorrt.HardwareCompatibilityLevel
Describes requirements of compatibility with GPU architectures other than that of the GPU on which the engine was
built. Levels except kNONE are only supported for engines built on NVIDIA Ampere and later GPUs.
Note that compatibility with future hardware depends on CUDA forward compatibility support.
Members:
NONE : Do not require hardware compatibility with GPU architectures other than that of the GPU
on which the engine was built.
AMPERE_PLUS : Require that the engine is compatible with Ampere and newer GPUs. This will
limit the max shared memory usage to 48KiB, may reduce the number of available tactics for each
layer, and may prevent some fusions from occurring. Thus this can decrease the performance,
especially for tf32 models. This option will disable cuDNN, cuBLAS, and cuBLAS LT as tactic
sources.
class tensorrt.IBuilderConfig
Variables
• min_timing_iterations - int [DEPRECATED] The number of minimization iterations
used when timing layers. When timing layers, the builder minimizes over a set of average
times for layer execution. This parameter controls the number of iterations used in minimiza-
tion. By default the minimum number of iterations is 1.
• avg_timing_iterations - int The number of averaging iterations used when timing lay-
ers. When timing layers, the builder minimizes over a set of average times for layer execution.
This parameter controls the number of iterations used in averaging. By default the number
of averaging iterations is 1.
• int8_calibrator - IInt8Calibrator Int8 Calibration interface. The calibrator is to
minimize the information loss during the INT8 quantization process.
• max_workspace_size - int [DEPRECATED] The maximum workspace size. The maxi-
mum GPU temporary memory which the engine can use at execution time.
• flags - int The build mode flags to turn on builder options for this network. The flags are
listed in the BuilderFlags enum. The flags set configuration options to build the network.
This should be in integer consisting of one or more BuilderFlag s, combined via binary
OR. For example, 1 << BuilderFlag.FP16 | 1 << BuilderFlag.DEBUG.
• profile_stream - int The handle for the CUDA stream that is used to profile this network.
• num_optimization_profiles - int The number of optimization profiles.
• default_device_type - tensorrt.DeviceType The default DeviceType to be used by
the Builder.
• DLA_core - int The DLA core that the engine executes on. Must be between 0 and N-1
where N is the number of available DLA cores.
16
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• profiling_verbosity - Profiling verbosity in NVTX annotations.
• engine_capability - The desired engine capability. See EngineCapability for details.
• algorithm_selector - The IAlgorithmSelector to use.
• builder_optimization_level - The builder optimization level which TensorRT should
build the engine at. Setting a higher optimization level allows TensorRT to spend longer
engine building time searching for more optimization options. The resulting engine may have
better performance compared to an engine built with a lower optimization level. The default
optimization level is 3. Valid values include integers from 0 to the maximum optimization
level, which is currently 5. Setting it to be greater than the maximum level results in identical
behavior to the maximum level.
• hardware_compatibility_level - Hardware compatibility allows an engine compatible
with GPU architectures other than that of the GPU on which the engine was built.
• plugins_to_serialize - The plugin libraries to be serialized with forward-compatible
engines.
• max_aux_streams - The maximum number of auxiliary streams that TRT is allowed to
use. If the network contains operators that can run in parallel, TRT can execute them us-
ing auxiliary streams in addition to the one provided to the IExecutionContext::enqueueV3()
call. The default maximum number of auxiliary streams is determined by the heuristics in
TensorRT on whether enabling multi-stream would improve the performance. This behavior
can be overridden by calling this API to set the maximum number of auxiliary streams ex-
plicitly. Set this to 0 to enforce single-stream inference. The resulting engine may use fewer
auxiliary streams than the maximum if the network does not contain enough parallelism or
if TensorRT determines that using more auxiliary streams does not help improve the per-
formance. Allowing more auxiliary streams does not always give better performance since
there will be synchronizations overhead between streams. Using CUDA graphs at runtime
can help reduce the overhead caused by cross-stream synchronizations. Using more auxil-
iary leads to more memory usage at runtime since some activation memory blocks will not
be able to be reused.
__del__(self: tensorrt.tensorrt.IBuilderConfig) → None
__exit__(exc_type, exc_value, traceback)
Context managers are deprecated and have no effect. Objects are automatically freed when the reference
count reaches 0.
__init__(*args, **kwargs)
add_optimization_profile(self: tensorrt.tensorrt.IBuilderConfig, profile:
tensorrt.tensorrt.IOptimizationProfile) → int
Add an optimization profile.
This function must be called at least once if the network has dynamic or shape input tensors.
Parameters profile - The new optimization profile, which must satisfy bool(profile) ==
True
Returns The index of the optimization profile (starting from 0) if the input is valid, or -1 if the
input is not valid.
can_run_on_DLA(self: tensorrt.tensorrt.IBuilderConfig, layer: tensorrt.tensorrt.ILayer) → bool
Check if the layer can run on DLA.
Parameters layer - The layer to check
Returns A bool indicating whether the layer can run on DLA
4.4. IBuilderConfig
17
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
clear_flag(self: tensorrt.tensorrt.IBuilderConfig, flag: tensorrt.tensorrt.BuilderFlag) → None
Clears the builder mode flag from the enabled flags.
Parameters flag - The flag to clear.
clear_quantization_flag(self: tensorrt.tensorrt.IBuilderConfig, flag:
tensorrt.tensorrt.QuantizationFlag) → None
Clears the quantization flag from the enabled quantization flags.
Parameters flag - The flag to clear.
create_timing_cache(self: tensorrt.tensorrt.IBuilderConfig, serialized_timing_cache: buffer) →
tensorrt.tensorrt.ITimingCache
Create timing cache
Create ITimingCache instance from serialized raw data. The created timing cache doesn’t belong to a
specific builder config. It can be shared by multiple builder instances
Parameters serialized_timing_cache - The serialized timing cache. If an empty cache is
provided (i.e. b""), a new cache will be created.
Returns The created ITimingCache object.
get_calibration_profile(self: tensorrt.tensorrt.IBuilderConfig) → tensorrt.tensorrt.IOptimizationProfile
Get the current calibration profile.
Returns The current calibration profile or nullptr if calibrartion profile is unset.
get_device_type(self: tensorrt.tensorrt.IBuilderConfig, layer: tensorrt.tensorrt.ILayer) →
tensorrt.tensorrt.DeviceType
Get the device that the layer executes on.
Parameters layer - The layer to get the DeviceType for
Returns The DeviceType of the layer
get_flag(self: tensorrt.tensorrt.IBuilderConfig, flag: tensorrt.tensorrt.BuilderFlag) → bool
Check if a build mode flag is set.
Parameters flag - The flag to check.
Returns A bool indicating whether the flag is set.
get_memory_pool_limit(self: tensorrt.tensorrt.IBuilderConfig, pool: tensorrt.tensorrt.MemoryPoolType)
→ int
Retrieve the memory size limit of the corresponding pool in bytes. If set_memory_pool_limit() for
the pool has not been called, this returns the default value used by TensorRT. This default value is not
necessarily the maximum possible value for that configuration.
Parameters pool - The memory pool to get the limit for.
Returns The size of the memory limit, in bytes, for the corresponding pool.
get_preview_feature(self: tensorrt.tensorrt.IBuilderConfig, feature: tensorrt.tensorrt.PreviewFeature) →
bool
Check if a preview feature is enabled.
Parameters feature - the feature to query
Returns true if the feature is enabled, false otherwise
get_quantization_flag(self: tensorrt.tensorrt.IBuilderConfig, flag: tensorrt.tensorrt.QuantizationFlag)
→ bool
Check if a quantization flag is set.
18
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Parameters flag - The flag to check.
Returns A bool indicating whether the flag is set.
get_tactic_sources(self: tensorrt.tensorrt.IBuilderConfig) → int
Get the tactic sources currently set in the engine build configuration.
get_timing_cache(self: tensorrt.tensorrt.IBuilderConfig) → tensorrt.tensorrt.ITimingCache
Get the timing cache from current IBuilderConfig
Returns The timing cache used in current IBuilderConfig, or None if no timing cache is set.
is_device_type_set(self: tensorrt.tensorrt.IBuilderConfig, layer: tensorrt.tensorrt.ILayer) → bool
Check if the DeviceType for a layer is explicitly set.
Parameters layer - The layer to check for DeviceType
Returns True if DeviceType is not default, False otherwise
reset(self: tensorrt.tensorrt.IBuilderConfig) → None
Resets the builder configuration to defaults. When initializing a builder config object, we can call this
function.
reset_device_type(self: tensorrt.tensorrt.IBuilderConfig, layer: tensorrt.tensorrt.ILayer) → None
Reset the DeviceType for the given layer.
Parameters layer - The layer to reset the DeviceType for
set_calibration_profile(self: tensorrt.tensorrt.IBuilderConfig, profile:
tensorrt.tensorrt.IOptimizationProfile) → bool
Set a calibration profile.
Calibration optimization profile must be set if int8 calibration is used to set scales for a network with runtime
dimensions.
Parameters profile - The new calibration profile, which must satisfy bool(profile) ==
True or be nullptr. MIN and MAX values will be overwritten by kOPT.
Returns True if the calibration profile was set correctly.
set_device_type(self: tensorrt.tensorrt.IBuilderConfig, layer: tensorrt.tensorrt.ILayer, device_type:
tensorrt.tensorrt.DeviceType) → None
Set the device that this layer must execute on. If DeviceType is not set or is reset, TensorRT will use the
default DeviceType set in the builder.
The DeviceType for a layer must be compatible with the safety flow (if specified). For example a layer
cannot be marked for DLA execution while the builder is configured for kSAFE_GPU.
Parameters
• layer - The layer to set the DeviceType of
• device_type - The DeviceType the layer must execute on
set_flag(self: tensorrt.tensorrt.IBuilderConfig, flag: tensorrt.tensorrt.BuilderFlag) → None
Add the input builder mode flag to the already enabled flags.
Parameters flag - The flag to set.
set_memory_pool_limit(self: tensorrt.tensorrt.IBuilderConfig, pool: tensorrt.tensorrt.MemoryPoolType,
pool_size: int) → None
Set the memory size for the memory pool.
TensorRT layers access different memory pools depending on the operation. This function sets in the
IBuilderConfig the size limit, specified by pool_size, for the corresponding memory pool, specified by
4.4. IBuilderConfig
19
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
pool. TensorRT will build a plan file that is constrained by these limits or report which constraint caused
the failure.
If the size of the pool, specified by pool_size, fails to meet the size requirements for the pool, this func-
tion does nothing and emits the recoverable error, ErrorCode.INVALID_ARGUMENT, to the registered
IErrorRecorder .
If the size of the pool is larger than the maximum possible value for the configuration, this function does
nothing and emits ErrorCode.UNSUPPORTED_STATE.
If the pool does not exist on the requested device type when building the network, a warning is emitted to
the logger, and the memory pool value is ignored.
Refer to MemoryPoolType to see the size requirements for each pool.
Parameters
• pool - The memory pool to limit the available memory for.
• pool_size - The size of the pool in bytes.
set_preview_feature(self: tensorrt.tensorrt.IBuilderConfig, feature: tensorrt.tensorrt.PreviewFeature,
enable: bool) → None
Enable or disable a specific preview feature.
Allows enabling or disabling experimental features, which are not enabled by default in the current release.
Preview Features have been fully tested but are not yet as stable as other features in TensorRT. They are
provided as opt-in features for at least one release.
Refer to PreviewFeature for additional information, and a list of the available features.
Parameters
• feature - the feature to enable
• enable - whether to enable or disable
set_quantization_flag(self: tensorrt.tensorrt.IBuilderConfig, flag: tensorrt.tensorrt.QuantizationFlag)
→ None
Add the input quantization flag to the already enabled quantization flags.
Parameters flag - The flag to set.
set_tactic_sources(self: tensorrt.tensorrt.IBuilderConfig, tactic_sources: int) → bool
Set tactic sources.
This bitset controls which tactic sources TensorRT is allowed to use for tactic selection.
Multiple tactic sources may be combined with a bitwise OR operation. For example, to enable cublas and
cublasLt as tactic sources, use a value of: 1 << int(trt.TacticSource.CUBLAS) | 1 << int(trt.
TacticSource.CUBLAS_LT)
Parameters tactic_sources - The tactic sources to set
Returns A bool indicating whether the tactic sources in the build configuration were updated.
The tactic sources in the build configuration will not be updated if the provided value is
invalid.
set_timing_cache(self: tensorrt.tensorrt.IBuilderConfig, cache: tensorrt.tensorrt.ITimingCache,
ignore_mismatch: bool) → bool
Attach a timing cache to IBuilderConfig
The timing cache has verification header to make sure the provided cache can be used in current environ-
ment. A failure will be reported if the CUDA device property in the provided cache is different from current
20
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
environment. bool(ignore_mismatch) == True skips strict verification and allows loading cache cre-
ated from a different device. The cache must not be destroyed until after the engine is built.
Parameters
• cache - The timing cache to be used
• ignore_mismatch - Whether or not allow using a cache that contains different CUDA
device property
Returns A BOOL indicating whether the operation is done successfully.
4.5 Builder
4.5.1 NetworkDefinitionCreationFlag
tensorrt.NetworkDefinitionCreationFlag
List of immutable network properties expressed at network creation time. For example, to enable ex-
plicit batch mode, pass a value of 1 << int(NetworkDefinitionCreationFlag.EXPLICIT_BATCH) to
create_network()
Members:
EXPLICIT_BATCH : Specify that the network should be created with an explicit batch dimension.
Creating a network without this flag has been deprecated.
EXPLICIT_PRECISION : [DEPRECATED] This flag has no effect now.
4.5.2 Builder
class tensorrt.Builder(self: tensorrt.tensorrt.Builder, logger: tensorrt.tensorrt.ILogger) → None
Builds an ICudaEngine from a INetworkDefinition .
Variables
• max_batch_size - int [DEPRECATED] For networks built with implicit batch, the max-
imum batch size which can be used at execution time, and also the batch size for which the
ICudaEngine will be optimized. This no effect for networks created with explicit batch
dimension mode.
• platform_has_tf32 - bool Whether the platform has tf32 support.
• platform_has_fast_fp16 - bool Whether the platform has fast native fp16.
• platform_has_fast_int8 - bool Whether the platform has fast native int8.
• max_DLA_batch_size - int The maximum batch size DLA can support. For any tensor the
total volume of index dimensions combined(dimensions other than CHW) with the requested
batch size should not exceed the value returned by this function.
• num_DLA_cores - int The number of DLA engines available to this builder.
• error_recorder - IErrorRecorder Application-implemented error reporting interface
for TensorRT objects.
• gpu_allocator - IGpuAllocator The GPU allocator to be used by the Builder . All
GPU memory acquired will use this allocator. If set to None, the default allocator will be
used.
4.5. Builder
21
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• logger - ILogger The logger provided when creating the refitter.
• max_threads - int The maximum thread that can be used by the Builder.
Parameters logger - The logger to use.
__del__(self: tensorrt.tensorrt.Builder) → None
__exit__(exc_type, exc_value, traceback)
Context managers are deprecated and have no effect. Objects are automatically freed when the reference
count reaches 0.
__init__(self: tensorrt.tensorrt.Builder, logger: tensorrt.tensorrt.ILogger) → None
Parameters logger - The logger to use.
build_engine(self: tensorrt.tensorrt.Builder, network: tensorrt.tensorrt.INetworkDefinition, config:
tensorrt.tensorrt.IBuilderConfig) → tensorrt.tensorrt.ICudaEngine
Builds an engine for the given INetworkDefinition and IBuilderConfig .
This enables the builder to build multiple engines based on the same network definition, but with different
builder configurations.
Parameters
• network - The TensorRT INetworkDefinition .
• config - The TensorRT IBuilderConfig .
Returns A new ICudaEngine .
build_serialized_network(self: tensorrt.tensorrt.Builder, network: tensorrt.tensorrt.INetworkDefinition,
config: tensorrt.tensorrt.IBuilderConfig) → tensorrt.tensorrt.IHostMemory
Builds and serializes a network for the given INetworkDefinition and IBuilderConfig .
This function allows building and serialization of a network without creating an engine.
Parameters
• network - Network definition.
• config - Builder configuration.
Returns A pointer to a IHostMemory object that contains a serialized network.
create_builder_config(self: tensorrt.tensorrt.Builder) → tensorrt.tensorrt.IBuilderConfig
Create a builder configuration object.
See IBuilderConfig
create_network(self: tensorrt.tensorrt.Builder, flags: int = 0) → tensorrt.tensorrt.INetworkDefinition
Create a INetworkDefinition object.
Parameters flags - NetworkDefinitionCreationFlag s combined using bitwise OR.
Please enable the NetworkDefinitionCreationFlag.EXPLICIT_BATCH flag whenever
possible.
Returns An empty TensorRT INetworkDefinition .
create_optimization_profile(self: tensorrt.tensorrt.Builder) → tensorrt.tensorrt.IOptimizationProfile
Create a new optimization profile.
If the network has any dynamic input tensors, the appropriate calls to IOptimizationProfile.
set_shape() must be made. Likewise, if there are any shape input tensors, the appropriate calls to
IOptimizationProfile.set_shape_input() are required.
22
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
See IOptimizationProfile
get_plugin_registry(self: tensorrt.tensorrt.Builder) → tensorrt.tensorrt.IPluginRegistry
Get the local plugin registry that can be used by the builder.
Returns The local plugin registry that can be used by the builder.
is_network_supported(self: tensorrt.tensorrt.Builder, network: tensorrt.tensorrt.INetworkDefinition,
config: tensorrt.tensorrt.IBuilderConfig) → bool
Checks that a network is within the scope of the IBuilderConfig settings.
Parameters
• network - The network definition to check for configuration compliance.
• config - The configuration of the builder to use when checking the network.
Given an INetworkDefinition and an IBuilderConfig , check if the network falls within the con-
straints of the builder configuration based on the EngineCapability , BuilderFlag , and DeviceType
Returns True if network is within the scope of the restrictions specified by the builder con-
fig, False otherwise. This function reports the conditions that are violated to the registered
ErrorRecorder .
NOTE: This function will synchronize the cuda stream returned by config.profile_stream before re-
turning.
reset(self: tensorrt.tensorrt.Builder) → None
Resets the builder state to default values.
4.6 ICudaEngine
tensorrt.TensorIOMode
IO tensor modes for TensorRT.
Members:
NONE : Tensor is not an input or output.
INPUT : Tensor is input to the engine.
OUTPUT : Tensor is output to the engine.
class tensorrt.ICudaEngine
An ICudaEngine for executing inference on a built network.
The engine can be indexed with [] . When indexed in this way with an integer, it will return the corresponding
binding name. When indexed with a string, it will return the corresponding binding index.
Variables
• num_bindings - int The number of binding indices.
• num_io_tensors - int The number of IO tensors.
• max_batch_size - int [DEPRECATED] The maximum batch size which can be used for
inference for an engine built from an INetworkDefinition with implicit batch dimension.
For an engine built from an INetworkDefinition with explicit batch dimension, this will
always be 1 .
4.6. ICudaEngine
23
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
•
has_implicit_batch_dimension - bool Whether the engine was built with an im-
plicit batch dimension. This is an engine-wide property. Either all tensors in the en-
gine have an implicit batch dimension or none of them do. This is True if and only
if the INetworkDefinition from which this engine was built was created without the
NetworkDefinitionCreationFlag.EXPLICIT_BATCH flag.
•
num_layers - int The number of layers in the network. The number of layers in the net-
work is not necessarily the number in the original INetworkDefinition, as layers may be
combined or eliminated as the ICudaEngine is optimized. This value can be useful when
building per-layer tables, such as when aggregating profiling data over a number of execu-
tions.
•
max_workspace_size - int The amount of workspace the ICudaEngine uses. The
workspace size will be no greater than the value provided to the Builder when the
ICudaEngine was built, and will typically be smaller. Workspace will be allocated for each
IExecutionContext .
•
device_memory_size - int The amount of device memory required by an
IExecutionContext .
•
refittable - bool Whether the engine can be refit.
•
name - str The name of the network associated with the engine. The name is set during
network creation and is retrieved after building or deserialization.
•
num_optimization_profiles - int The number of optimization profiles defined for this
engine. This is always at least 1.
•
error_recorder - IErrorRecorder Application-implemented error reporting interface
for TensorRT objects.
•
engine_capability
- EngineCapability The engine capability.
See
EngineCapability for details.
•
tactic_sources - int The tactic sources required by this engine.
•
profiling_verbosity - The profiling verbosity the builder config was set to when the
engine was built.
•
hardware_compatibility_level - The hardware compatibility level of the engine.
•
num_aux_streams - Read-only. The number of auxiliary streams used by this engine, which
will be less than or equal to the maximum allowed number of auxiliary streams by setting
builder_config.max_aux_streams when the engine is built.
__del__(self: tensorrt.tensorrt.ICudaEngine) → None
__exit__(exc_type, exc_value, traceback)
Context managers are deprecated and have no effect. Objects are automatically freed when the reference
count reaches 0.
__getitem__(*args, **kwargs)
Overloaded function.
1.
__getitem__(self: tensorrt.tensorrt.ICudaEngine, arg0: str) -> int
2.
__getitem__(self: tensorrt.tensorrt.ICudaEngine, arg0: int) -> str
__init__(*args, **kwargs)
__len__(self: tensorrt.tensorrt.ICudaEngine) → int
binding_is_input(*args, **kwargs)
Overloaded function.
24
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
1. binding_is_input(self: tensorrt.tensorrt.ICudaEngine, index: int) -> bool
Determine whether a binding is an input binding.
index The binding index.
returns True if the index corresponds to an input binding and the index is in range.
2. binding_is_input(self: tensorrt.tensorrt.ICudaEngine, name: str) -> bool
Determine whether a binding is an input binding.
name The name of the tensor corresponding to an engine binding.
returns True if the index corresponds to an input binding and the index is in range.
create_engine_inspector(self: tensorrt.tensorrt.ICudaEngine) → nvinfer1::IEngineInspector
Create an IEngineInspector which prints out the layer information of an engine or an execution context.
Returns The IEngineInspector.
create_execution_context(self: tensorrt.tensorrt.ICudaEngine) → tensorrt.tensorrt.IExecutionContext
Create an IExecutionContext .
Returns The newly created IExecutionContext .
create_execution_context_without_device_memory(self: tensorrt.tensorrt.ICudaEngine) →
tensorrt.tensorrt.IExecutionContext
Create an IExecutionContext without any device memory allocated The memory for execution of this
device context must be supplied by the application.
Returns An IExecutionContext without device memory allocated.
get_binding_bytes_per_component(self: tensorrt.tensorrt.ICudaEngine, index: int) → int
Return the number of bytes per component of an element. The vector component size is returned if
get_binding_vectorized_dim() != -1.
Parameters index - The binding index.
get_binding_components_per_element(self: tensorrt.tensorrt.ICudaEngine, index: int) → int
Return the number of components included in one element.
The number of elements in the vectors is returned if get_binding_vectorized_dim() != -1.
Parameters index - The binding index.
get_binding_dtype(*args, **kwargs)
Overloaded function.
1. get_binding_dtype(self: tensorrt.tensorrt.ICudaEngine, index: int) -> tensorrt.tensorrt.DataType
Determine the required data type for a buffer from its binding index.
index The binding index.
Returns The type of data in the buffer.
2. get_binding_dtype(self: tensorrt.tensorrt.ICudaEngine, name: str) -> tensorrt.tensorrt.DataType
Determine the required data type for a buffer from its binding index.
name The name of the tensor corresponding to an engine binding.
Returns The type of data in the buffer.
get_binding_format(self: tensorrt.tensorrt.ICudaEngine, index: int) → tensorrt.tensorrt.TensorFormat
Return the binding format.
4.6. ICudaEngine
25
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Parameters index - The binding index.
get_binding_format_desc(self: tensorrt.tensorrt.ICudaEngine, index: int) → str
Return the human readable description of the tensor format.
The description includes the order, vectorization, data type, strides, etc. For example:
Example 1: kCHW + FP32
“Row major linear FP32 format”
Example 2: kCHW2 + FP16
“Two wide channel vectorized row major FP16 format”
Example 3: kHWC8 + FP16 + Line Stride = 32
“Channel major FP16 format where C % 8 == 0 and H Stride % 32 == 0”
Parameters index - The binding index.
get_binding_index(self: tensorrt.tensorrt.ICudaEngine, name: str) → int
Retrieve the binding index for a named tensor.
You can also use engine’s __getitem__() with engine[name]. When invoked with a str , this will
return the corresponding binding index.
IExecutionContext.execute_async_v2() and IExecutionContext.execute_v2() require an ar-
ray of buffers. Engine bindings map from tensor names to indices in this array. Binding indices are assigned
at ICudaEngine build time, and take values in the range [0 . . . n-1] where n is the total number of inputs
and outputs.
Parameters name - The tensor name.
Returns The binding index for the named tensor, or -1 if the name is not found.
get_binding_name(self: tensorrt.tensorrt.ICudaEngine, index: int) → str
Retrieve the name corresponding to a binding index.
You can also use engine’s __getitem__() with engine[index]. When invoked with an int , this will
return the corresponding binding name.
This is the reverse mapping to that provided by get_binding_index() .
Parameters index - The binding index.
Returns The name corresponding to the binding index.
get_binding_shape(*args, **kwargs)
Overloaded function.
1. get_binding_shape(self: tensorrt.tensorrt.ICudaEngine, index: int) -> tensorrt.tensorrt.Dims
Get the shape of a binding.
index The binding index.
Returns The shape of the binding if the index is in range, otherwise Dims()
2. get_binding_shape(self: tensorrt.tensorrt.ICudaEngine, name: str) -> tensorrt.tensorrt.Dims
Get the shape of a binding.
name The name of the tensor corresponding to an engine binding.
Returns The shape of the binding if the tensor is present, otherwise Dims()
26
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
get_binding_vectorized_dim(self: tensorrt.tensorrt.ICudaEngine, index: int) → int
Return the dimension index that the buffer is vectorized.
Specifically -1 is returned if scalars per vector is 1.
Parameters index - The binding index.
get_location(*args, **kwargs)
Overloaded function.
1. get_location(self: tensorrt.tensorrt.ICudaEngine, index: int) -> tensorrt.tensorrt.TensorLocation
Get location of binding. This lets you know whether the binding should be a pointer to device
or host memory.
index The binding index.
returns The location of the bound tensor with given index.
2. get_location(self: tensorrt.tensorrt.ICudaEngine, name: str) -> tensorrt.tensorrt.TensorLocation
Get location of binding. This lets you know whether the binding should be a pointer to device
or host memory.
name The name of the tensor corresponding to an engine binding.
returns The location of the bound tensor with given index.
get_profile_shape(*args, **kwargs)
Overloaded function.
1. get_profile_shape(self:
tensorrt.tensorrt.ICudaEngine, profile_index:
int, binding:
int)
->
List[tensorrt.tensorrt.Dims]
Get the minimum/optimum/maximum dimensions for a particular binding under an optimiza-
tion profile.
arg profile_index The index of the profile.
arg binding The binding index or name.
returns A List[Dims] of length 3, containing the minimum, optimum, and maximum
shapes, in that order.
2. get_profile_shape(self:
tensorrt.tensorrt.ICudaEngine, profile_index:
int, binding:
str)
->
List[tensorrt.tensorrt.Dims]
Get the minimum/optimum/maximum dimensions for a particular binding under an optimiza-
tion profile.
arg profile_index The index of the profile.
arg binding The binding index or name.
returns A List[Dims] of length 3, containing the minimum, optimum, and maximum
shapes, in that order.
get_profile_shape_input(*args, **kwargs)
Overloaded function.
1. get_profile_shape_input(self: tensorrt.tensorrt.ICudaEngine, profile_index: int, binding: int)
->
List[List[int]]
Get minimum/optimum/maximum values for an input shape binding under an optimization
profile. If the specified binding is not an input shape binding, an exception is raised.
arg profile_index The index of the profile.
4.6. ICudaEngine
27
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
arg binding The binding index or name.
returns A List[List[int]] of length 3, containing the minimum, optimum, and max-
imum values, in that order. If the values have not been set yet, an empty list is returned.
2. get_profile_shape_input(self: tensorrt.tensorrt.ICudaEngine, profile_index: int, binding: str) ->
List[List[int]]
Get minimum/optimum/maximum values for an input shape binding under an optimization
profile. If the specified binding is not an input shape binding, an exception is raised.
arg profile_index The index of the profile.
arg binding The binding index or name.
returns A List[List[int]] of length 3, containing the minimum, optimum, and max-
imum values, in that order. If the values have not been set yet, an empty list is returned.
get_tensor_bytes_per_component(*args, **kwargs)
Overloaded function.
1. get_tensor_bytes_per_component(self: tensorrt.tensorrt.ICudaEngine, name: str) -> int
Return the number of bytes per component of an element.
The vector component size is returned if get_tensor_vectorized_dim() != -1.
arg name The tensor name.
2. get_tensor_bytes_per_component(self: tensorrt.tensorrt.ICudaEngine, name: str, profile_index: int)
-> int
Return the number of bytes per component of an element.
The vector component size is returned if get_tensor_vectorized_dim() != -1.
arg name The tensor name.
get_tensor_components_per_element(*args, **kwargs)
Overloaded function.
1. get_tensor_components_per_element(self: tensorrt.tensorrt.ICudaEngine, name: str) -> int
Return the number of components included in one element.
The number of elements in the vectors is returned if get_tensor_vectorized_dim() !=
-1.
arg name The tensor name.
2. get_tensor_components_per_element(self: tensorrt.tensorrt.ICudaEngine, name: str, profile_index:
int) -> int
Return the number of components included in one element.
The number of elements in the vectors is returned if get_tensor_vectorized_dim() !=
-1.
arg name The tensor name.
get_tensor_dtype(self: tensorrt.tensorrt.ICudaEngine, name: str) → tensorrt.tensorrt.DataType
Return the required data type for a buffer from its tensor name.
Parameters name - The tensor name.
get_tensor_format(*args, **kwargs)
Overloaded function.
28
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
1. get_tensor_format(self: tensorrt.tensorrt.ICudaEngine, name: str) -> tensorrt.tensorrt.TensorFormat
Return the tensor format.
arg name The tensor name.
2. get_tensor_format(self: tensorrt.tensorrt.ICudaEngine, name: str, profile_index: int) -> ten-
sorrt.tensorrt.TensorFormat
Return the tensor format.
arg name The tensor name.
get_tensor_format_desc(*args, **kwargs)
Overloaded function.
1. get_tensor_format_desc(self: tensorrt.tensorrt.ICudaEngine, name: str) -> str
Return the human readable description of the tensor format.
The description includes the order, vectorization, data type, strides, etc. For example:
Example 1: kCHW + FP32
“Row major linear FP32 format”
Example 2: kCHW2 + FP16
“Two wide channel vectorized row major FP16 format”
Example 3: kHWC8 + FP16 + Line Stride = 32
“Channel major FP16 format where C % 8 == 0 and H Stride % 32 == 0”
arg name The tensor name.
2. get_tensor_format_desc(self: tensorrt.tensorrt.ICudaEngine, name: str, profile_index: int) -> str
Return the human readable description of the tensor format.
The description includes the order, vectorization, data type, strides, etc. For example:
Example 1: kCHW + FP32
“Row major linear FP32 format”
Example 2: kCHW2 + FP16
“Two wide channel vectorized row major FP16 format”
Example 3: kHWC8 + FP16 + Line Stride = 32
“Channel major FP16 format where C % 8 == 0 and H Stride % 32 == 0”
arg name The tensor name.
get_tensor_location(self: tensorrt.tensorrt.ICudaEngine, name: str) → tensorrt.tensorrt.TensorLocation
Determine whether an input or output tensor must be on GPU or CPU.
Parameters name - The tensor name.
get_tensor_mode(self: tensorrt.tensorrt.ICudaEngine, name: str) → nvinfer1::TensorIOMode
Determine whether a tensor is an input or output tensor.
Parameters name - The tensor name.
4.6. ICudaEngine
29
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
get_tensor_name(self: tensorrt.tensorrt.ICudaEngine, index: int) → str
Return the name of an input or output tensor.
Parameters index - The tensor index.
get_tensor_profile_shape(self: tensorrt.tensorrt.ICudaEngine, name: str, profile_index: int) →
List[tensorrt.tensorrt.Dims]
Get the minimum/optimum/maximum dimensions for a particular tensor under an optimization profile.
Parameters
• name - The tensor name.
• profile_index - The index of the profile.
get_tensor_shape(self: tensorrt.tensorrt.ICudaEngine, name: str) → tensorrt.tensorrt.Dims
Return the shape of an input or output tensor.
Parameters name - The tensor name.
get_tensor_vectorized_dim(*args, **kwargs)
Overloaded function.
1. get_tensor_vectorized_dim(self: tensorrt.tensorrt.ICudaEngine, name: str) -> int
Return the dimension index that the buffer is vectorized.
Specifically -1 is returned if scalars per vector is 1.
arg name The tensor name.
2. get_tensor_vectorized_dim(self: tensorrt.tensorrt.ICudaEngine, name: str, profile_index: int) -> int
Return the dimension index that the buffer is vectorized.
Specifically -1 is returned if scalars per vector is 1.
arg name The tensor name.
is_execution_binding(self: tensorrt.tensorrt.ICudaEngine, binding: int) → bool
Returns True if tensor is required for execution phase, false otherwise.
For example, if a network uses an input tensor with binding i ONLY as the reshape dimensions for an
IShuffleLayer , then is_execution_binding(i) == False, and a binding of 0 can be supplied for
it when calling IExecutionContext.execute_v2() or IExecutionContext.execute_async_v2()
Parameters binding - The binding index.
is_shape_binding(self: tensorrt.tensorrt.ICudaEngine, binding: int) → bool
Returns True if tensor is required as input for shape calculations or output from them.
TensorRT evaluates a network in two phases:
1. Compute shape information required to determine memory allocation requirements and validate that
runtime sizes make sense.
2. Process tensors on the device.
Some tensors are required in phase 1. These tensors are called “shape tensors”, and always have type
tensorrt.int32 and no more than one dimension. These tensors are not always shapes themselves, but
might be used to calculate tensor shapes for phase 2.
is_shape_binding() returns true if the tensor is a required input or an output computed in phase 1.
is_execution_binding() returns true if the tensor is a required input or an output computed in phase 2.
30
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
For example, if a network uses an input tensor with binding i as an input to an IElementWiseLayer that
computes the reshape dimensions for an IShuffleLayer , is_shape_binding(i) == True
It’s possible to have a tensor be required by both phases. For instance, a tensor can be used as a shape in
an IShuffleLayer and as the indices for an IGatherLayer collecting floating-point data.
It’s also possible to have a tensor required by neither phase that shows up in the engine’s inputs. For example,
if an input tensor is used only as an input to an IShapeLayer , only its shape matters and its values are
irrelevant.
Parameters binding - The binding index.
is_shape_inference_io(self: tensorrt.tensorrt.ICudaEngine, name: str) → bool
Determine whether a tensor is read or written by infer_shapes.
Parameters name - The tensor name.
serialize(self: tensorrt.tensorrt.ICudaEngine) → tensorrt.tensorrt.IHostMemory
Serialize the engine to a stream.
Returns An IHostMemory object containing the serialized ICudaEngine .
4.7 IExecutionContext
class tensorrt.IOutputAllocator(self: tensorrt.tensorrt.IOutputAllocator) → None
Application-implemented class for controlling output tensor allocation.
To implement a custom output allocator, ensure that you explicitly instantiate the base class in __init__() :
class MyOutputAllocator(trt.IOutputAllocator):
def __init__(self):
trt.IOutputAllocator.__init__(self)
def reallocate_output(self, tensor_name, memory, size, alignment):
... # Your implementation here
def notify_shape(self, tensor_name, shape):
... # Your implementation here
__init__(self: tensorrt.tensorrt.IOutputAllocator) → None
notify_shape(self: tensorrt.tensorrt.IOutputAllocator, tensor_name: str, shape: tensorrt.tensorrt.Dims) →
None
Called by TensorRT when the shape of the output tensor is known.
Parameters
• tensor_name - The output tensor name.
• shape - The output tensor shape.
reallocate_output(self: tensorrt.tensorrt.IOutputAllocator, tensor_name: str, memory: capsule, size: int,
alignment: int) → capsule
A callback implemented by the application to handle acquisition of output tensor memory.
If an allocation request cannot be satisfied, None should be returned.
Parameters
• tensor_name - The output tensor name.
4.7. IExecutionContext
31
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• memory - The output tensor memory address.
• size - The number of bytes required.
• alignment - The required alignment of memory.
Returns The address of the output tensor memory.
class tensorrt.IExecutionContext
Context for executing inference using an ICudaEngine . Multiple IExecutionContext s may exist for one
ICudaEngine instance, allowing the same ICudaEngine to be used for the execution of multiple batches simul-
taneously.
Variables
•
debug_sync - bool The debug sync flag. If this flag is set to true, the ICudaEngine will
log the successful execution for each kernel during execute_v2(). It has no effect when using
execute_async_v2().
•
profiler - IProfiler The profiler in use by this IExecutionContext .
•
engine - ICudaEngine The associated ICudaEngine .
•
name - str The name of the IExecutionContext .
•
device_memory - capsule The device memory for use by this execution context. The
memory must be aligned on a 256-byte boundary, and its size must be at least engine.
device_memory_size. If using execute_async_v2() to run the network, The memory
is in use from the invocation of execute_async_v2() until network execution is complete.
If using execute_v2(), it is in use until execute_v2() returns. Releasing or otherwise
using the memory for other purposes during this time will result in undefined behavior.
•
active_optimization_profile - int The active optimization profile for the con-
text.
The selected profile will be used in subsequent calls to execute_v2() or
execute_async_v2() . Profile 0 is selected by default. Changing this value will invalidate
all dynamic bindings for the current execution context, so that they have to be set again using
set_binding_shape() before calling either execute_v2() or execute_async_v2() .
•
all_binding_shapes_specified - bool Whether all dynamic dimensions of input ten-
sors have been specified by calling set_binding_shape() . Trivially true if network
has no dynamically shaped input tensors. Does not work with name-base interfaces eg.
set_input_shape(). Use infer_shapes() instead.
•
all_shape_inputs_specified - bool Whether values for all input shape tensors have
been specified by calling set_shape_input() . Trivially true if network has no input
shape bindings. Does not work with name-base interfaces eg. set_input_shape(). Use
infer_shapes() instead.
•
error_recorder - IErrorRecorder Application-implemented error reporting interface
for TensorRT objects.
•
enqueue_emits_profile - bool Whether enqueue emits layer timing to the profiler. The
default value is True. If set to False, enqueue will be asynchronous if there is a profiler
attached. An extra method IExecutionContext::report_to_profiler() needs to be
called to obtain the profiling data and report to the profiler attached.
•
persistent_cache_limit - The maximum size of persistent L2 cache that this execution
context may use for activation caching. Activation caching is not supported on all architec-
tures - see “How TensorRT uses Memory” in the developer guide for details. The default is
0 Bytes.
32
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• nvtx_verbosity - The NVTX verbosity of the execution context. Building with kDE-
TAILED verbosity will generally increase latency in enqueueV2/V3(). Call this method to
select NVTX verbosity in this execution context at runtime. The default is the verbosity
with which the engine was built, and the verbosity may not be raised above that level. This
function does not affect how IEngineInspector interacts with the engine.
• temporary_allocator - IGpuAllocator The GPU allocator used for internal temporary
storage.
__del__(self: tensorrt.tensorrt.IExecutionContext) → None
__exit__(exc_type, exc_value, traceback)
Context managers are deprecated and have no effect. Objects are automatically freed when the reference
count reaches 0.
__init__(*args, **kwargs)
execute(self: tensorrt.tensorrt.IExecutionContext, batch_size: int = 1, bindings: List[int]) → bool
[DEPRECATED] Please use execute_v2() instead if the engine is built from a network with explicit batch
dimension mode enabled.
Synchronously execute inference on a batch. This method requires a array of input and output buffers. The
mapping from tensor names to indices can be queried using ICudaEngine.get_binding_index() .
Parameters
• batch_size - The batch size. This is at most the value supplied when the ICudaEngine
was built. This has no effect if the engine is built from a network with explicit batch di-
mension mode enabled.
• bindings - A list of integers representing input and output buffer addresses for the net-
work.
Returns True if execution succeeded.
execute_async(self: tensorrt.tensorrt.IExecutionContext, batch_size: int = 1, bindings: List[int],
stream_handle: int, input_consumed: capsule = None) → bool
[DEPRECATED] Please use execute_async_v2() instead if the engine is built from a network with explicit
batch dimension mode enabled.
Asynchronously execute inference on a batch. This method requires a array of input and output buffers.
The mapping from tensor names to indices can be queried using ICudaEngine::get_binding_index()
Parameters
• batch_size - The batch size. This is at most the value supplied when the ICudaEngine
was built. This has no effect if the engine is built from a network with explicit batch di-
mension mode enabled.
• bindings - A list of integers representing input and output buffer addresses for the net-
work.
• stream_handle - A handle for a CUDA stream on which the inference kernels will be
executed.
• input_consumed - An optional event which will be signaled when the input buffers can
be refilled with new data
Returns True if the kernels were executed successfully.
4.7. IExecutionContext
33
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
execute_async_v2(self: tensorrt.tensorrt.IExecutionContext, bindings: List[int], stream_handle: int,
input_consumed: capsule = None) → bool
Asynchronously execute inference on a batch. This method requires a array of input and output buffers.
The mapping from tensor names to indices can be queried using ICudaEngine::get_binding_index()
. This method only works for execution contexts built from networks with no implicit batch dimension.
Parameters
• bindings - A list of integers representing input and output buffer addresses for the net-
work.
• stream_handle - A handle for a CUDA stream on which the inference kernels will be
executed.
• input_consumed - An optional event which will be signaled when the input buffers can
be refilled with new data
Returns True if the kernels were executed successfully.
execute_async_v3(self: tensorrt.tensorrt.IExecutionContext, stream_handle: int) → bool
Asynchronously execute inference.
Modifying or releasing memory that has been registered for the tensors before stream synchronization or
the event passed to set_input_consumed_event() has been triggered results in undefined behavior.
Input tensors can be released after the set_input_consumed_event() whereas output tensors require
stream synchronization.
Parameters stream_handle - The cuda stream on which the inference kernels will be en-
queued.
execute_v2(self: tensorrt.tensorrt.IExecutionContext, bindings: List[int]) → bool
Synchronously execute inference on a batch. This method requires a array of input and output buffers. The
mapping from tensor names to indices can be queried using ICudaEngine.get_binding_index() . This
method only works for execution contexts built from networks with no implicit batch dimension.
Parameters bindings - A list of integers representing input and output buffer addresses for the
network.
Returns True if execution succeeded.
get_binding_shape(self: tensorrt.tensorrt.IExecutionContext, binding: int) → tensorrt.tensorrt.Dims
Get the dynamic shape of a binding.
If set_binding_shape() has been called on this binding (or if there are no dynamic dimensions),
all dimensions will be positive. Otherwise, it is necessary to call set_binding_shape() before
execute_async_v2() or execute_v2() may be called.
If the binding is out of range, an invalid Dims with nbDims == -1 is returned.
If
ICudaEngine.binding_is_input(binding)
is
False
,
then
both
all_binding_shapes_specified and all_shape_inputs_specified must be True before
calling this method.
Parameters binding - The binding index.
Returns A Dims object representing the currently selected shape.
get_input_consumed_event(self: tensorrt.tensorrt.IExecutionContext) → int
Return the event associated with consuming the input tensors.
get_max_output_size(self: tensorrt.tensorrt.IExecutionContext, name: str) → int
Return the upper bound on an output tensor’s size, in bytes, based on the current optimization profile.
34
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
If the profile or input shapes are not yet set, or the provided name does not map to an output, returns -1.
Parameters name - The tensor name.
get_output_allocator(self: tensorrt.tensorrt.IExecutionContext, name: str) → nvinfer1::IOutputAllocator
Return the output allocator associated with given output tensor, or None if the provided name does not map
to an output tensor.
Parameters name - The tensor name.
get_shape(self: tensorrt.tensorrt.IExecutionContext, binding: int) → List[int]
Get values of an input shape tensor required for shape calculations or an output tensor produced by shape
calculations.
Parameters binding - The binding index of an input tensor for which ICudaEngine.
is_shape_binding(binding) is true.
If
ICudaEngine.binding_is_input(binding) == False,
then
both
all_binding_shapes_specified and all_shape_inputs_specified must be True before
calling this method.
Returns An iterable containing the values of the shape tensor.
get_strides(self: tensorrt.tensorrt.IExecutionContext, binding: int) → tensorrt.tensorrt.Dims
Return the strides of the buffer for the given binding.
Note that strides can be different for different execution contexts with dynamic shapes.
Parameters binding - The binding index.
get_tensor_address(self: tensorrt.tensorrt.IExecutionContext, name: str) → int
Get memory address for the given input or output tensor.
Parameters name - The tensor name.
get_tensor_shape(self: tensorrt.tensorrt.IExecutionContext, name: str) → tensorrt.tensorrt.Dims
Return the shape of the given input or output tensor.
Parameters name - The tensor name.
get_tensor_strides(self: tensorrt.tensorrt.IExecutionContext, name: str) → tensorrt.tensorrt.Dims
Return the strides of the buffer for the given tensor name.
Note that strides can be different for different execution contexts with dynamic shapes.
Parameters name - The tensor name.
infer_shapes(self: tensorrt.tensorrt.IExecutionContext) → List[str]
Infer shapes and return the names of any tensors that are insufficiently specified.
An input tensor is insufficiently specified if either of the following is true:
• It has dynamic dimensions and its runtime dimensions have not yet been specified via
set_input_shape() .
• is_shape_inference_io(t) is True and the tensor’s address has not yet been set.
Returns A List[str] indicating the names of any tensors which have not been sufficiently
specified, or an empty list on success.
Raises RuntimeError if shape inference fails due to reasons other than insufficiently specified
tensors.
4.7. IExecutionContext
35
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
report_to_profiler(self: tensorrt.tensorrt.IExecutionContext) → bool
Calculate layer timing info for the current optimization profile in IExecutionContext and update the profiler
after one iteration of inference launch.
If the enqueue_emits_profiler flag was set to true, the enqueue function will calculate layer timing implicitly
if a profiler is provided. There is no need to call this function. If the enqueue_emits_profiler flag was set
to false, the enqueue function will record the CUDA event timers if a profiler is provided. But it will not
perform the layer timing calculation. This function needs to be called explicitly to calculate layer timing
for the previous inference launch.
In the CUDA graph launch scenario, it will record the same set of CUDA events as in regular enqueue
functions if the graph is captured from an IExecutionContext with profiler enabled. This function needs
to be called after graph launch to report the layer timing info to the profiler.
Profiling CUDA graphs is only available from CUDA 11.1 onwards.
Returns True if the call succeeded, else False (e.g. profiler not provided, in CUDA graph
capture mode, etc.)
set_aux_streams(self: tensorrt.tensorrt.IExecutionContext, aux_streams: List[int]) → None
Set the auxiliary streams that TensorRT should launch kernels on in the next execute_async_v3() call.
If set, TensorRT will launch the kernels that are supposed to run on the auxiliary streams using the streams
provided by the user with this API. If this API is not called before the execute_async_v3() call, then Ten-
sorRT will use the auxiliary streams created by TensorRT internally.
TensorRT will always insert event synchronizations between the main stream provided via execute_async_v3() call an
• At the beginning of the execute_async_v3() call, TensorRT will make sure that all the auxiliary
streams wait on the activities on the main stream.
• At the end of the execute_async_v3() call, TensorRT will make sure that the main stream wait on
the activities on all the auxiliary streams.
The provided auxiliary streams must not be the default stream and must all be different to avoid deadlocks.
Parameters aux_streams - A list of cuda streams. If the length of the list is greater than en-
gine.num_aux_streams, then only the first “engine.num_aux_streams” streams will be used.
If the length is less than engine.num_aux_streams, such as an empty list, then TensorRT will
use the provided streams for the first few auxiliary streams, and will create additional streams
internally for the rest of the auxiliary streams.
set_binding_shape(self: tensorrt.tensorrt.IExecutionContext, binding: int, shape: tensorrt.tensorrt.Dims)
→ bool
Set the dynamic shape of a binding.
Requires the engine to be built without an implicit batch dimension. The binding must be an input tensor,
and all dimensions must be compatible with the network definition (i.e. only the wildcard dimension -1
can be replaced with a new dimension > 0). Furthermore, the dimensions must be in the valid range for the
currently selected optimization profile.
For all dynamic non-output bindings (which have at least one wildcard dimension of
-1),
this method needs to be called after setting active_optimization_profile before either
execute_async_v2() or execute_v2() may be called. When all input shapes have been speci-
fied, all_binding_shapes_specified is set to True .
Parameters
• binding - The binding index.
• shape - The shape to set.
36
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Returns False if an error occurs (e.g. specified binding is out of range for the currently selected
optimization profile or specified shape is inconsistent with min-max range of the optimization
profile), else True.
Note that the network can still be invalid for certain combinations of input shapes that lead to invalid output
shapes. To confirm the correctness of the network input shapes, check whether the output binding has valid
shape using get_binding_shape() on the output binding.
set_input_consumed_event(self: tensorrt.tensorrt.IExecutionContext, event: int) → bool
Mark all input tensors as consumed.
Parameters event - The cuda event that is triggered after all input tensors have been consumed.
set_input_shape(self: tensorrt.tensorrt.IExecutionContext, name: str, shape: tensorrt.tensorrt.Dims) →
bool
Set shape for the given input tensor.
Parameters
• name - The input tensor name.
• shape - The input tensor shape.
set_optimization_profile_async(self: tensorrt.tensorrt.IExecutionContext, profile_index: int,
stream_handle: int) → bool
Set the optimization profile with async semantics
Parameters
• profile_index - The index of the optimization profile
• stream_handle - cuda stream on which the work to switch optimization profile can be
enqueued
When an optimization profile is switched via this API, TensorRT may require that data is copied via cud-
aMemcpyAsync. It is the application’s responsibility to guarantee that synchronization between the profile
sync stream and the enqueue stream occurs.
Returns True if the optimization profile was set successfully
set_output_allocator(self: tensorrt.tensorrt.IExecutionContext, name: str, output_allocator:
nvinfer1::IOutputAllocator) → bool
Set output allocator to use for the given output tensor.
Pass None to unset the output allocator.
The allocator is called by execute_async_v3().
Parameters
• name - The tensor name.
• output_allocator - The output allocator.
set_shape_input(self: tensorrt.tensorrt.IExecutionContext, binding: int, shape: List[int]) → bool
Set values of an input shape tensor required by shape calculations.
Parameters
• binding - The binding index of an input tensor for which ICudaEngine.
is_shape_binding(binding) and ICudaEngine.binding_is_input(binding)
are both true.
• shape - An iterable containing the values of the input shape tensor. The number of values
should be the product of the dimensions returned by get_binding_shape(binding).
4.7. IExecutionContext
37
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
If ICudaEngine.is_shape_binding(binding) and ICudaEngine.binding_is_input(binding)
are both true, this method must be called before execute_async_v2() or execute_v2() may be called.
Additionally, this method must not be called if either ICudaEngine.is_shape_binding(binding) or
ICudaEngine.binding_is_input(binding) are false.
Returns False if an error occurs (e.g. specified binding is out of range for the currently selected
optimization profile or specified shape values are inconsistent with min-max range of the
optimization profile), else True.
Note that the network can still be invalid for certain combinations of input shapes that lead to invalid output
shapes. To confirm the correctness of the network input shapes, check whether the output binding has valid
shape using get_binding_shape() on the output binding.
set_tensor_address(self: tensorrt.tensorrt.IExecutionContext, name: str, memory: int) → bool
Set memory address for the given input or output tensor.
Parameters
• name - The tensor name.
• memory - The memory address.
4.8 Runtime
tensorrt.TempfileControlFlag
Flags used to control TensorRT’s behavior when creating executable temporary files.
On some platforms the TensorRT runtime may need to create files in a temporary directory or use platform-
specific APIs to create files in-memory to load temporary DLLs that implement runtime code. These flags allow
the application to explicitly control TensorRT’s use of these files. This will preclude the use of certain TensorRT
APIs for deserializing and loading lean runtimes.
These should be treated as bit offsets, e.g. in order to allow in-memory files for a given IRuntime:
runtime.tempfile_control_flags |= (1 << int(TempfileControlFlag.ALLOW_IN_MEMORY_
˓→
FILES))
Members:
ALLOW_IN_MEMORY_FILES : Allow creating and loading files in-memory (or unnamed files).
ALLOW_TEMPORARY_FILES : Allow creating and loading named files in a temporary directory
on the filesystem.
class tensorrt.Runtime(self: tensorrt.tensorrt.Runtime, logger: tensorrt.tensorrt.ILogger) → None
Allows a serialized ICudaEngine to be deserialized.
Variables
• error_recorder - IErrorRecorder Application-implemented error reporting interface
for TensorRT objects.
• gpu_allocator - IGpuAllocator The GPU allocator to be used by the Runtime . All
GPU memory acquired will use this allocator. If set to None, the default allocator will be
used (Default: cudaMalloc/cudaFree).
• DLA_core - int The DLA core that the engine executes on. Must be between 0 and N-1
where N is the number of available DLA cores.
• num_DLA_cores - int The number of DLA engines available to this builder.
38
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• logger - ILogger The logger provided when creating the refitter.
• max_threads - int The maximum thread that can be used by the Runtime.
• temporary_directory - str The temporary directory to use when loading executable
code for engines. If set to None (the default), TensorRT will attempt to find a suitable direc-
tory for use using platform-specific heuristics: - On UNIX/Linux platforms, TensorRT will
first try the TMPDIR environment variable, then fall back to /tmp - On Windows, TensorRT
will try the TEMP environment variable.
• tempfile_control_flags - int Flags which control whether TensorRT is allowed to
create in-memory or temporary files. See TempfileControlFlag for details.
• engine_host_code_allowed - bool Whether this runtime is allowed to deserialize en-
gines that contain host executable code (Default: False).
Parameters logger - The logger to use.
__del__(self: tensorrt.tensorrt.Runtime) → None
__exit__(exc_type, exc_value, traceback)
Context managers are deprecated and have no effect. Objects are automatically freed when the reference
count reaches 0.
__init__(self: tensorrt.tensorrt.Runtime, logger: tensorrt.tensorrt.ILogger) → None
Parameters logger - The logger to use.
deserialize_cuda_engine(self: tensorrt.tensorrt.Runtime, serialized_engine: buffer) →
tensorrt.tensorrt.ICudaEngine
Deserialize an ICudaEngine from a stream.
Parameters serialized_engine - The buffer that holds the serialized ICudaEngine .
Returns The ICudaEngine, or None if it could not be deserialized.
get_plugin_registry(self: tensorrt.tensorrt.Runtime) → tensorrt.tensorrt.IPluginRegistry
Get the local plugin registry that can be used by the runtime.
Returns The local plugin registry that can be used by the runtime.
load_runtime(self: tensorrt.tensorrt.Runtime, path: str) → tensorrt.tensorrt.Runtime
Load IRuntime from the file.
This method loads a runtime library from a shared library file.
The runtime can then
be used to execute a plan file built with BuilderFlag.VERSION_COMPATIBLE and Builder-
Flag.EXCLUDE_LEAN_RUNTIME both set and built with the same version of TensorRT as the loaded
runtime library.
Variables path - Path to the runtime lean library.
Returns The IRuntime, or None if it could not be loaded.
4.8. Runtime
39
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
4.9 Refitter
class tensorrt.Refitter(self: tensorrt.tensorrt.Refitter, engine: tensorrt.tensorrt.ICudaEngine, logger:
tensorrt.tensorrt.ILogger) → None
Updates weights in an ICudaEngine .
Variables
• error_recorder - IErrorRecorder Application-implemented error reporting interface
for TensorRT objects.
• logger - ILogger The logger provided when creating the refitter.
• max_threads - int The maximum thread that can be used by the Refitter.
Parameters
• engine - The engine to refit.
• logger - The logger to use.
__del__(self: tensorrt.tensorrt.Refitter) → None
__exit__(exc_type, exc_value, traceback)
Context managers are deprecated and have no effect. Objects are automatically freed when the reference
count reaches 0.
__init__(self: tensorrt.tensorrt.Refitter, engine: tensorrt.tensorrt.ICudaEngine, logger:
tensorrt.tensorrt.ILogger) → None
Parameters
• engine - The engine to refit.
• logger - The logger to use.
get_all(self: tensorrt.tensorrt.Refitter) → Tuple[List[str], List[tensorrt.tensorrt.WeightsRole]]
Get description of all weights that could be refitted.
Returns The names of layers with refittable weights, and the roles of those weights.
get_all_weights(self: tensorrt.tensorrt.Refitter) → List[str]
Get names of all weights that could be refitted.
Returns The names of refittable weights.
get_dynamic_range(self: tensorrt.tensorrt.Refitter, tensor_name: str) → tuple
Gets the dynamic range of a tensor. If the dynamic range was never set, returns the range computed during
calibration.
Parameters tensor_name - The name of the tensor whose dynamic range to retrieve.
Returns Tuple[float, float] A tuple containing the [minimum, maximum] of the dynamic
range.
get_missing(self: tensorrt.tensorrt.Refitter) → Tuple[List[str], List[tensorrt.tensorrt.WeightsRole]]
Get description of missing weights.
For example, if some Weights have been set, but the engine was optimized in a way that combines weights,
any unsupplied Weights in the combination are considered missing.
Returns The names of layers with missing weights, and the roles of those weights.
40
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
get_missing_weights(self: tensorrt.tensorrt.Refitter) → List[str]
Get names of missing weights.
For example, if some Weights have been set, but the engine was optimized in a way that combines weights,
any unsupplied Weights in the combination are considered missing.
Returns The names of missing weights, empty string for unnamed weights.
get_tensors_with_dynamic_range(self: tensorrt.tensorrt.Refitter) → List[str]
Get names of all tensors that have refittable dynamic ranges.
Returns The names of tensors with refittable dynamic ranges.
refit_cuda_engine(self: tensorrt.tensorrt.Refitter) → bool
Updates associated engine. Return True if successful.
Failure occurs if get_missing() != 0 before the call.
set_dynamic_range(self: tensorrt.tensorrt.Refitter, tensor_name: str, range: List[float]) → bool
Update dynamic range for a tensor.
Parameters
• tensor_name - The name of the tensor whose dynamic range to update.
• range - The new range.
Returns True if successful, False otherwise.
Returns false if there is no Int8 engine tensor derived from a network tensor of that name. If successful,
then get_missing() may report that some weights need to be supplied.
set_named_weights(self: tensorrt.tensorrt.Refitter, name: str, weights: tensorrt.tensorrt.Weights) → bool
Specify new weights of given name. Possible reasons for rejection are:
• The name of weights is empty or does not correspond to any refittable weights.
• The number of weights is inconsistent with the original specification.
Modifying the weights before method refit_cuda_engine() completes will result in undefined behavior.
Parameters
• name - The name of the weights to be refitted.
• weights - The new weights to associate with the name.
Returns True on success, or False if new weights are rejected.
set_weights(self: tensorrt.tensorrt.Refitter, layer_name: str, role: tensorrt.tensorrt.WeightsRole, weights:
tensorrt.tensorrt.Weights) → bool
Specify new weights for a layer of given name. Possible reasons for rejection are:
• There is no such layer by that name.
• The layer does not have weights with the specified role.
• The number of weights is inconsistent with the layer’s original specification.
Modifying the weights before refit_cuda_engine() completes will result in undefined behavior.
Parameters
• layer_name - The name of the layer.
• role - The role of the weights. See WeightsRole for more information.
• weights - The weights to refit with.
4.9. Refitter
41
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Returns True on success, or False if new weights are rejected.
4.10 IErrorRecorder
tensorrt.ErrorCodeTRT
Error codes that can be returned by TensorRT during execution.
Members:
SUCCESS : Execution completed successfully.
UNSPECIFIED_ERROR : An error that does not fall into any other category. This error is included
for forward compatibility.
INTERNAL_ERROR : A non-recoverable TensorRT error occurred.
INVALID_ARGUMENT : An argument passed to the function is invalid in isolation. This is a
violation of the API contract.
INVALID_CONFIG : An error occurred when comparing the state of an argument relative to other
arguments. For example, the dimensions for concat differ between two tensors outside of the
channel dimension. This error is triggered when an argument is correct in isolation, but not
relative to other arguments. This is to help to distinguish from the simple errors from the more
complex errors. This is a violation of the API contract.
FAILED_ALLOCATION : An error occurred when performing an allocation of memory on the
host or the device. A memory allocation error is normally fatal, but in the case where the appli-
cation provided its own memory allocation routine, it is possible to increase the pool of available
memory and resume execution.
FAILED_INITIALIZATION : One, or more, of the components that TensorRT relies on did not
initialize correctly. This is a system setup issue.
FAILED_EXECUTION : An error occurred during execution that caused TensorRT to end pre-
maturely, either an asynchronous error or other execution errors reported by CUDA/DLA. In a
dynamic system, the data can be thrown away and the next frame can be processed or execution
can be retried. This is either an execution error or a memory error.
FAILED_COMPUTATION : An error occurred during execution that caused the data to become
corrupted, but execution finished. Examples of this error are NaN squashing or integer over-
flow. In a dynamic system, the data can be thrown away and the next frame can be processed or
execution can be retried. This is either a data corruption error, an input error, or a range error.
INVALID_STATE : TensorRT was put into a bad state by incorrect sequence of function calls. An
example of an invalid state is specifying a layer to be DLA only without GPU fallback, and that
layer is not supported by DLA. This can occur in situations where a service is optimistically
executing networks for multiple different configurations without checking proper error configu-
rations, and instead throwing away bad configurations caught by TensorRT. This is a violation of
the API contract, but can be recoverable.
Example of a recovery: GPU fallback is disabled and conv layer with large filter(63x63) is spec-
ified to run on DLA. This will fail due to DLA not supporting the large kernel size. This can be
recovered by either turning on GPU fallback or setting the layer to run on the GPU.
UNSUPPORTED_STATE : An error occurred due to the network not being supported on the device
due to constraints of the hardware or system. An example is running a unsafe layer in a safety
certified context, or a resource requirement for the current network is greater than the capabilities
of the target device. The network is otherwise correct, but the network and hardware combination
is problematic. This can be recoverable. Examples: * Scratch space requests larger than available
42
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
device memory and can be recovered by increasing allowed workspace size. * Tensor size exceeds
the maximum element count and can be recovered by reducing the maximum batch size.
class tensorrt.IErrorRecorder(self: tensorrt.tensorrt.IErrorRecorder) → None
Reference counted application-implemented error reporting interface for TensorRT objects.
The error reporting mechanism is a user defined object that interacts with the internal state of the object that it is
assigned to in order to determine information about abnormalities in execution. The error recorder gets both an
error enum that is more descriptive than pass/fail and also a description that gives more detail on the exact failure
modes. In the safety context, the error strings are all limited to 128 characters in length. The ErrorRecorder gets
passed along to any class that is created from another class that has an ErrorRecorder assigned to it. For example,
assigning an ErrorRecorder to an Builder allows all INetwork’s, ILayer’s, and ITensor’s to use the same error
recorder. For functions that have their own ErrorRecorder accessor functions. This allows registering a different
error recorder or de-registering of the error recorder for that specific object.
The ErrorRecorder object implementation must be thread safe if the same ErrorRecorder is passed to different
interface objects being executed in parallel in different threads. All locking and synchronization is pushed to
the interface implementation and TensorRT does not hold any synchronization primitives when accessing the
interface functions.
clear(self: tensorrt.tensorrt.IErrorRecorder) → None
Clear the error stack on the error recorder.
Removes all the tracked errors by the error recorder. This function must guarantee that after this function
is called, and as long as no error occurs, num_errors will be zero.
get_error_code(self: tensorrt.tensorrt.IErrorRecorder, arg0: int) → tensorrt.tensorrt.ErrorCodeTRT
Returns the ErrorCode enumeration.
The error_idx specifies what error code from 0 to num_errors-1 that the application wants to analyze and
return the error code enum.
Parameters error_idx - A 32bit integer that indexes into the error array.
Returns Returns the enum corresponding to error_idx.
get_error_desc(self: tensorrt.tensorrt.IErrorRecorder, arg0: int) → str
Returns description of the error.
For the error specified by the idx value, return description of the error. In the safety context there is a
constant length requirement to remove any dynamic memory allocations and the error message may be
truncated. The format of the error description is “<EnumAsStr> - <Description>”.
Parameters error_idx - A 32bit integer that indexes into the error array.
Returns Returns description of the error.
has_overflowed(self: tensorrt.tensorrt.IErrorRecorder) → bool
Determine if the error stack has overflowed.
In the case when the number of errors is large, this function is used to query if one or more errors have been
dropped due to lack of storage capacity. This is especially important in the automotive safety case where
the internal error handling mechanisms cannot allocate memory.
Returns True if errors have been dropped due to overflowing the error stack.
num_errors(self: tensorrt.tensorrt.IErrorRecorder) → int
Return the number of errors
Determines the number of errors that occurred between the current point in execution and the last time
that the clear() was executed. Due to the possibility of asynchronous errors occuring, a TensorRT API
can return correct results, but still register errors with the Error Recorder. The value of getNbErrors must
monotonically increases until clear() is called.
4.10. IErrorRecorder
43
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Returns Returns the number of errors detected, or 0 if there are no errors.
report_error(self: tensorrt.tensorrt.IErrorRecorder, arg0: tensorrt.tensorrt.ErrorCodeTRT, arg1: str) →
bool
Clear the error stack on the error recorder.
Report an error to the user that has a given value and human readable description. The function returns
false if processing can continue, which implies that the reported error is not fatal. This does not guarantee
that processing continues, but provides a hint to TensorRT.
Parameters
• val - The error code enum that is being reported.
• desc - The description of the error.
Returns True if the error is determined to be fatal and processing of the current function must
end.
4.11 ITimingCache
class tensorrt.ITimingCache
Class to handle tactic timing info collected from builder.
combine(self: tensorrt.tensorrt.ITimingCache, input_cache: tensorrt.tensorrt.ITimingCache,
ignore_mismatch: bool) → bool
Combine input timing cache into local instance.
Append entries in input cache to local cache. Conflicting entries will be skipped. The input cache must be
generated by a TensorRT build of exact same version, otherwise combine will be skipped and return false.
bool(ignore_mismatch) == True if combining a timing cache created from a different device.
Parameters
• input_cache - The input timing cache
• ignore_mismatch - Whether or not to allow cache verification header mismatch
Returns A bool indicating whether the combine operation is done successfully.
reset(self: tensorrt.tensorrt.ITimingCache) → bool
Empty the timing cache
Returns A bool indicating whether the reset operation is done successfully.
serialize(self: tensorrt.tensorrt.ITimingCache) → tensorrt.tensorrt.IHostMemory
Serialize a timing cache to a IHostMemory object.
Returns An IHostMemory object that contains a serialized timing cache.
44
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
4.12 GPU Allocator
4.12.1 AllocatorFlag
tensorrt.AllocatorFlag
Members:
RESIZABLE : TensorRT may call realloc() on this allocation
4.12.2 IGpuAllocator
class tensorrt.IGpuAllocator(self: tensorrt.tensorrt.IGpuAllocator) → None
Application-implemented class for controlling allocation on the GPU.
To implement a custom allocator, ensure that you explicitly instantiate the base class in __init__() :
class MyAllocator(trt.IGpuAllocator):
def __init__(self):
trt.IGpuAllocator.__init__(self)
__init__(self: tensorrt.tensorrt.IGpuAllocator) → None
allocate(self: tensorrt.tensorrt.IGpuAllocator, size: int, alignment: int, flags: int) → capsule
A callback implemented by the application to handle acquisition of GPU memory. If an allocation request
of size 0 is made, None should be returned.
If an allocation request cannot be satisfied, None should be returned.
Parameters
• size - The size of the memory required.
• alignment - The required alignment of memory. Alignment will be zero or a power
of 2 not exceeding the alignment guaranteed by cudaMalloc. Thus this allocator can be
safely implemented with cudaMalloc/cudaFree. An alignment value of zero indicates any
alignment is acceptable.
• flags - Allocation flags. See AllocatorFlag
Returns The address of the allocated memory
deallocate(self: tensorrt.tensorrt.IGpuAllocator, memory: capsule) → bool
A callback implemented by the application to handle release of GPU memory.
TensorRT may pass a 0 to this function if it was previously returned by allocate().
Parameters memory - The memory address of the memory to release.
Returns True if the acquired memory is released successfully.
free(self: tensorrt.tensorrt.IGpuAllocator, memory: capsule) → None
A callback implemented by the application to handle release of GPU memory.
TensorRT may pass a 0 to this function if it was previously returned by allocate().
Parameters memory - The memory address of the memory to release.
4.12. GPU Allocator
45
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
reallocate(self: tensorrt.tensorrt.IGpuAllocator, address: capsule, alignment: int, new_size: int) →
capsule
A callback implemented by the application to resize an existing allocation.
Only allocations which were allocated with AllocatorFlag.RESIZABLE will be resized.
Options are one of: - resize in place leaving min(old_size, new_size) bytes unchanged and return the origi-
nal address - move min(old_size, new_size) bytes to a new location of sufficient size and return its address
- return nullptr, to indicate that the request could not be fulfilled.
If nullptr is returned, TensorRT will assume that resize() is not implemented, and that the allocation at
address is still valid.
This method is made available for use cases where delegating the resize strategy to the applica-
tion provides an opportunity to improve memory management. One possible implementation is to
allocate a large virtual device buffer and progressively commit physical memory with cuMemMap.
CU_MEM_ALLOC_GRANULARITY_RECOMMENDED is suggested in this case.
TensorRT may call realloc to increase the buffer by relatively small amounts.
Parameters
• address - the address of the original allocation.
• alignment - The alignment used by the original allocation.
• new_size - The new memory size required.
Returns The address of the reallocated memory
4.13 EngineInspector
class tensorrt.EngineInspector
An engine inspector which prints out the layer information of an engine or an execution context. The engine or
the context must be set before get_layer_information() or get_engine_information() can be called.
The amount of printed information depends on the profiling verbosity setting of the builder config when the
engine is built. By default, the profiling verbosity is set to ProfilingVerbosity.LAYER_NAMES_ONLY, and only
layer names will be printed. If the profiling verbosity is set to ProfilingVerbosity.DETAILED, layer names and
layer parameters will be printed. If the profiling verbosity is set to ProfilingVerbosity.NONE, no layer information
will be printed.
Variables
• engine - ICudaEngine Set or get the engine currently being inspected.
• context - IExecutionContext Set or get context currently being inspected.
• error_recorder - IErrorRecorder Application-implemented error reporting interface
for TensorRT objects.
__init__(*args, **kwargs)
get_engine_information(self: tensorrt.tensorrt.EngineInspector, format:
tensorrt.tensorrt.LayerInformationFormat) → str
Get a string describing the information about all the layers in the current engine or the execution context.
Parameters format - LayerInformationFormat The format the layer information should be
printed in.
Returns A string describing the information about all the layers in the current engine or the
execution context.
46
Chapter 4. Core
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
get_layer_information(self: tensorrt.tensorrt.EngineInspector, layer_index: int, format:
tensorrt.tensorrt.LayerInformationFormat) → str
Get a string describing the information about a specific layer in the current engine or the execution context.
Parameters
• layer_index - The index of the layer. It must lie in [0, engine.num_layers].
• format - LayerInformationFormat The format the layer information should be printed
in.
Returns A string describing the information about a specific layer in the current engine or the
execution context.
4.13. EngineInspector
47
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
48
Chapter 4. Core
CHAPTER
FIVE
NETWORK
5.1 INetworkDefinition
class tensorrt.INetworkDefinition
Represents a TensorRT Network from which the Builder can build an Engine
Variables
• num_layers - int The number of layers in the network.
• num_inputs - int The number of inputs of the network.
• num_outputs - int The number of outputs of the network.
• name - str The name of the network. This is used so that it can be associated with a built
engine. The name must be at most 128 characters in length. TensorRT makes no use of this
string except storing it as part of the engine so that it may be retrieved at runtime. A name
unique to the builder will be generated by default.
• has_implicit_batch_dimension - bool Whether the network was created with an
implicit batch dimension.
This is a network-wide property.
Either all tensors in
the network have an implicit batch dimension or none of them do. This is True
when the INetworkDefinition is created with default flags: create_network(). To
specify explicit batch, set the flag: create_network(flags=1 << int(tensorrt.
NetworkDefinitionCreationFlag.EXPLICIT_BATCH)).
• has_explicit_precision - bool True if and only if this INetworkDefinition
was
created
with
NetworkDefinitionCreationFlag.EXPLICIT_PRECISION
set:
create_network(flags=(1 << int(NetworkDefinitionCreationFlag.
EXPLICIT_PRECISION))).
• error_recorder - IErrorRecorder Application-implemented error reporting interface
for TensorRT objects.
__del__(self: tensorrt.tensorrt.INetworkDefinition) → None
__exit__(exc_type, exc_value, traceback)
Context managers are deprecated and have no effect. Objects are automatically freed when the reference
count reaches 0.
__getitem__(self: tensorrt.tensorrt.INetworkDefinition, arg0: int) → tensorrt.tensorrt.ILayer
__init__(*args, **kwargs)
__len__(self: tensorrt.tensorrt.INetworkDefinition) → int
49
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
add_activation(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, type:
tensorrt.tensorrt.ActivationType) → tensorrt.tensorrt.IActivationLayer
Add an activation layer to the network. See IActivationLayer for more information.
Parameters
• input - The input tensor to the layer.
• type - The type of activation function to apply.
Returns The new activation layer, or None if it could not be created.
add_assertion(self: tensorrt.tensorrt.INetworkDefinition, condition: tensorrt.tensorrt.ITensor, message:
str) → tensorrt.tensorrt.IAssertionLayer
Add a assertion layer. See IAssertionLayer for more information.
Parameters
• condition - The condition tensor to the layer.
• message - The message to print if the assertion fails.
Returns The new assertion layer, or None if it could not be created.
add_cast(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, to_type:
tensorrt.tensorrt.DataType) → tensorrt.tensorrt.ICastLayer
Add a cast layer. See ICastLayer for more information.
Parameters
• input - The input tensor to the layer.
• to_type - The data type the output tensor should be cast into.
Returns The new cast layer, or None if it could not be created.
add_concatenation(self: tensorrt.tensorrt.INetworkDefinition, inputs: List[tensorrt.tensorrt.ITensor]) →
tensorrt.tensorrt.IConcatenationLayer
Add a concatenation layer to the network. Note that all tensors must have the same dimension except for
the Channel dimension. See IConcatenationLayer for more information.
Parameters inputs - The input tensors to the layer.
Returns The new concatenation layer, or None if it could not be created.
add_constant(self: tensorrt.tensorrt.INetworkDefinition, shape: tensorrt.tensorrt.Dims, weights:
tensorrt.tensorrt.Weights) → tensorrt.tensorrt.IConstantLayer
Add a constant layer to the network. See IConstantLayer for more information.
Parameters
• shape - The shape of the constant.
• weights - The constant value, represented as weights.
Returns The new constant layer, or None if it could not be created.
add_convolution(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor,
num_output_maps: int, kernel_shape: tensorrt.tensorrt.DimsHW, kernel:
tensorrt.tensorrt.Weights, bias: tensorrt.tensorrt.Weights = None) →
tensorrt.tensorrt.IConvolutionLayer
Add a 2D convolution layer to the network. See IConvolutionLayer for more information.
Parameters
• input - The input tensor to the convolution.
50
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• num_output_maps - The number of output feature maps for the convolution.
• kernel_shape - The dimensions of the convolution kernel.
• kernel - The kernel weights for the convolution.
• bias - The optional bias weights for the convolution.
Returns The new convolution layer, or None if it could not be created.
add_convolution_nd(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor,
num_output_maps: int, kernel_shape: tensorrt.tensorrt.Dims, kernel:
tensorrt.tensorrt.Weights, bias: tensorrt.tensorrt.Weights = None) →
tensorrt.tensorrt.IConvolutionLayer
Add a multi-dimension convolution layer to the network. See IConvolutionLayer for more information.
Parameters
• input - The input tensor to the convolution.
• num_output_maps - The number of output feature maps for the convolution.
• kernel_shape - The dimensions of the convolution kernel.
• kernel - The kernel weights for the convolution.
• bias - The optional bias weights for the convolution.
Returns The new convolution layer, or None if it could not be created.
add_deconvolution(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor,
num_output_maps: int, kernel_shape: tensorrt.tensorrt.DimsHW, kernel:
tensorrt.tensorrt.Weights, bias: tensorrt.tensorrt.Weights = None) →
tensorrt.tensorrt.IDeconvolutionLayer
Add a 2D deconvolution layer to the network. See IDeconvolutionLayer for more information.
Parameters
• input - The input tensor to the layer.
• num_output_maps - The number of output feature maps.
• kernel_shape - The dimensions of the convolution kernel.
• kernel - The kernel weights for the convolution.
• bias - The optional bias weights for the convolution.
Returns The new deconvolution layer, or None if it could not be created.
add_deconvolution_nd(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor,
num_output_maps: int, kernel_shape: tensorrt.tensorrt.Dims, kernel:
tensorrt.tensorrt.Weights, bias: tensorrt.tensorrt.Weights = None) →
tensorrt.tensorrt.IDeconvolutionLayer
Add a multi-dimension deconvolution layer to the network. See IDeconvolutionLayer for more infor-
mation.
Parameters
• input - The input tensor to the layer.
• num_output_maps - The number of output feature maps.
• kernel_shape - The dimensions of the convolution kernel.
• kernel - The kernel weights for the convolution.
5.1. INetworkDefinition
51
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• bias - The optional bias weights for the convolution.
Returns The new deconvolution layer, or None if it could not be created.
add_dequantize(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, scale:
tensorrt.tensorrt.ITensor) → tensorrt.tensorrt.IDequantizeLayer
Add a dequantization layer to the network. See IDequantizeLayer for more information.
Parameters
• input - A tensor to quantize.
• scale - A tensor with the scale coefficients.
Returns The new dequantization layer, or None if it could not be created.
add_einsum(self: tensorrt.tensorrt.INetworkDefinition, inputs: List[tensorrt.tensorrt.ITensor], equation: str)
→ tensorrt.tensorrt.IEinsumLayer
Adds an Einsum layer to the network. See IEinsumLayer for more information.
Parameters
• inputs - The input tensors to the layer.
• equation - The Einsum equation of the layer.
Returns the new Einsum layer, or None if it could not be created.
add_elementwise(self: tensorrt.tensorrt.INetworkDefinition, input1: tensorrt.tensorrt.ITensor, input2:
tensorrt.tensorrt.ITensor, op: tensorrt.tensorrt.ElementWiseOperation) →
tensorrt.tensorrt.IElementWiseLayer
Add an elementwise layer to the network. See IElementWiseLayer for more information.
Parameters
• input1 - The first input tensor to the layer.
• input2 - The second input tensor to the layer.
• op - The binary operation that the layer applies.
The input tensors must have the same number of dimensions. For each dimension, their lengths must match,
or one of them must be one. In the latter case, the tensor is broadcast along that axis.
The output tensor has the same number of dimensions as the inputs. For each dimension, its length is the
maximum of the lengths of the corresponding input dimension.
Returns The new element-wise layer, or None if it could not be created.
add_fill(self: tensorrt.tensorrt.INetworkDefinition, shape: tensorrt.tensorrt.Dims, op:
tensorrt.tensorrt.FillOperation) → tensorrt.tensorrt.IFillLayer
Add a fill layer. See IFillLayer for more information.
Parameters
• dimensions - The output tensor dimensions.
• op - The fill operation that the layer applies.
Returns The new fill layer, or None if it could not be created.
add_fully_connected(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor,
num_outputs: int, kernel: tensorrt.tensorrt.Weights, bias: tensorrt.tensorrt.Weights
= None) → tensorrt.tensorrt.IFullyConnectedLayer
Add a fully connected layer to the network. See IFullyConnectedLayer for more information.
Parameters
52
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• input - The input tensor to the layer.
• num_outputs - The number of outputs of the layer.
• kernel - The kernel weights for the convolution.
• bias - The optional bias weights for the convolution.
Returns The new fully connected layer, or None if it could not be created.
add_gather(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, indices:
tensorrt.tensorrt.ITensor, axis: int) → tensorrt.tensorrt.IGatherLayer
Add a gather layer to the network. See IGatherLayer for more information.
Parameters
• input - The tensor to gather values from.
• indices - The tensor to get indices from to populate the output tensor.
• axis - The non-batch dimension axis in the data tensor to gather on.
Returns The new gather layer, or None if it could not be created.
add_gather_v2(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, indices:
tensorrt.tensorrt.ITensor, mode: tensorrt.tensorrt.GatherMode) →
tensorrt.tensorrt.IGatherLayer
Add a gather layer to the network. See IGatherLayer for more information.
Parameters
• input - The tensor to gather values from.
• indices - The tensor to get indices from to populate the output tensor.
• mode - The gather mode.
Returns The new gather layer, or None if it could not be created.
add_grid_sample(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, grid:
tensorrt.tensorrt.ITensor) → tensorrt.tensorrt.IGridSampleLayer
Creates a GridSample layer with a trt.InterpolationMode.LINEAR, unaligned corners, and
trt.SampleMode.FILL for 4d-shape input tensors. See IGridSampleLayer for more information.
Parameters
• input - The input tensor to the layer.
• grid - The grid tensor to the layer.
Variables
• interpolation_mode - class:InterpolationMode The interpolation mode to use in the
layer. Default is LINEAR.
• align_corners - class:bool the align mode to use in the layer. Default is False.
• padding_mode - SampleMode The padding mode to use in the layer. Default is FILL.
Returns The new grid sample layer, or None if it could not be created.
add_identity(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.IIdentityLayer
Add an identity layer. See IIdentityLayer for more information.
Parameters input - The input tensor to the layer.
Returns The new identity layer, or None if it could not be created.
5.1. INetworkDefinition
53
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
add_if_conditional(self: tensorrt.tensorrt.INetworkDefinition) → tensorrt.tensorrt.IIfConditional
Adds an if-conditional to the network, which provides a way to specify subgraphs that will be conditionally
executed using lazy evaluation. See IIfConditional for more information.
Returns The new if-condtional, or None if it could not be created.
add_input(self: tensorrt.tensorrt.INetworkDefinition, name: str, dtype: tensorrt.tensorrt.DataType, shape:
tensorrt.tensorrt.Dims) → tensorrt.tensorrt.ITensor
Adds an input to the network.
Parameters
• name - The name of the tensor.
• dtype - The data type of the tensor. Currently, tensorrt.int8 is not supported for inputs.
• shape - The dimensions of the tensor. The total volume must be less than 2^30 elements.
Returns The newly added Tensor.
add_loop(self: tensorrt.tensorrt.INetworkDefinition) → tensorrt.tensorrt.ILoop
Adds a loop to the network, which provides a way to specify a recurrent subgraph. See ILoop for more
information.
Returns The new loop layer, or None if it could not be created.
add_lrn(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, window: int, alpha: float,
beta: float, k: float) → tensorrt.tensorrt.ILRNLayer
Add a LRN layer to the network. See ILRNLayer for more information.
Parameters
• input - The input tensor to the layer.
• window - The size of the window.
• alpha - The alpha value for the LRN computation.
• beta - The beta value for the LRN computation.
• k - The k value for the LRN computation.
Returns The new LRN layer, or None if it could not be created.
add_matrix_multiply(self: tensorrt.tensorrt.INetworkDefinition, input0: tensorrt.tensorrt.ITensor, op0:
tensorrt.tensorrt.MatrixOperation, input1: tensorrt.tensorrt.ITensor, op1:
tensorrt.tensorrt.MatrixOperation) → tensorrt.tensorrt.IMatrixMultiplyLayer
Add a matrix multiply layer to the network. See IMatrixMultiplyLayer for more information.
Parameters
• input0 - The first input tensor (commonly A).
• op0 - Whether to treat input0 as matrices, transposed matrices, or vectors.
• input1 - The second input tensor (commonly B).
• op1 - Whether to treat input1 as matrices, transposed matrices, or vectors.
Returns The new matrix multiply layer, or None if it could not be created.
add_nms(self: tensorrt.tensorrt.INetworkDefinition, boxes: tensorrt.tensorrt.ITensor, scores:
tensorrt.tensorrt.ITensor, max_output_boxes_per_class: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.INMSLayer
Add a non-maximum suppression layer to the network. See INMSLayer for more information.
Parameters
54
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• boxes - The input boxes tensor to the layer.
• scores - The input scores tensor to the layer.
• max_output_boxes_per_class - The maxOutputBoxesPerClass tensor to the layer.
Variables
• bounding_box_format - BoundingBoxFormat The bounding box format used by the
layer. Default is CORNER_PAIRS.
• topk_box_limit - int The maximum number of filtered boxes considered for selection
per batch item. Default is 2000 for SM 5.3 and 6.2 devices, and 5000 otherwise. The TopK
box limit must be less than or equal to {2000 for SM 5.3 and 6.2 devices, 5000 otherwise}.
Returns The new NMS layer, or None if it could not be created.
add_non_zero(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.INonZeroLayer
Adds an NonZero layer to the network. See INonZeroLayer for more information.
Parameters input - The input tensor to the layer.
Returns the new NonZero layer, or None if it could not be created.
add_normalization(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, scale:
tensorrt.tensorrt.ITensor, bias: tensorrt.tensorrt.ITensor, axesMask: int) →
tensorrt.tensorrt.INormalizationLayer
Adds a Normalization layer to the network. See Normalization for more information.
Parameters
• input - The input tensor to the layer.
• scale - The scale tensor used to scale the normalized output.
• bias - The bias tensor used to scale the normalized output.
• axesMask - The axes on which to perform mean calculations. The bit in position i of
bitmask axes corresponds to explicit dimension i of the result. E.g., the least significant bit
corresponds to the first explicit dimension and the next to least significant bit corresponds
to the second explicit dimension.
Returns the new Normalization layer, or None if it could not be created.
add_one_hot(self: tensorrt.tensorrt.INetworkDefinition, indices: tensorrt.tensorrt.ITensor, values:
tensorrt.tensorrt.ITensor, depth: tensorrt.tensorrt.ITensor, axis: int) →
tensorrt.tensorrt.IOneHotLayer
Add a OneHot layer to the network. See IOneHotLayer for more information.
Parameters
• indices - The tensor to get indices from to populate the output tensor.
• values - The tensor to get off (cold) value and on (hot) value
• depth - The tensor to get depth (number of classes) of one-hot encoding
• axis - The axis to append the one-hot encoding to
Returns The new OneHot layer, or None if it could not be created.
add_padding(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, pre_padding:
tensorrt.tensorrt.DimsHW, post_padding: tensorrt.tensorrt.DimsHW) →
tensorrt.tensorrt.IPaddingLayer
Add a 2D padding layer to the network. See IPaddingLayer for more information.
5.1. INetworkDefinition
55
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Parameters
• input - The input tensor to the layer.
• pre_padding - The padding to apply to the start of the tensor.
• post_padding - The padding to apply to the end of the tensor.
Returns The new padding layer, or None if it could not be created.
add_padding_nd(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, pre_padding:
tensorrt.tensorrt.Dims, post_padding: tensorrt.tensorrt.Dims) →
tensorrt.tensorrt.IPaddingLayer
Add a multi-dimensional padding layer to the network. See IPaddingLayer for more information.
Parameters
• input - The input tensor to the layer.
• pre_padding - The padding to apply to the start of the tensor.
• post_padding - The padding to apply to the end of the tensor.
Returns The new padding layer, or None if it could not be created.
add_parametric_relu(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, slopes:
tensorrt.tensorrt.ITensor) → tensorrt.tensorrt.IParametricReLULayer
Add a parametric ReLU layer. See IParametricReLULayer for more information.
Parameters
• input - The input tensor to the layer.
• slopes - The slopes tensor (input elements are multiplied with the slopes where the input
is negative).
Returns The new parametric ReLU layer, or None if it could not be created.
add_plugin_v2(self: tensorrt.tensorrt.INetworkDefinition, inputs: List[tensorrt.tensorrt.ITensor], plugin:
tensorrt.tensorrt.IPluginV2) → tensorrt.tensorrt.IPluginV2Layer
Add a plugin layer to the network using an IPluginV2 interface. See IPluginV2 for more information.
Parameters
• inputs - The input tensors to the layer.
• plugin - The layer plugin.
Returns The new plugin layer, or None if it could not be created.
add_pooling(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, type:
tensorrt.tensorrt.PoolingType, window_size: tensorrt.tensorrt.DimsHW) →
tensorrt.tensorrt.IPoolingLayer
Add a 2D pooling layer to the network. See IPoolingLayer for more information.
Parameters
• input - The input tensor to the layer.
• type - The type of pooling to apply.
• window_size - The size of the pooling window.
Returns The new pooling layer, or None if it could not be created.
56
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
add_pooling_nd(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, type:
tensorrt.tensorrt.PoolingType, window_size: tensorrt.tensorrt.Dims) →
tensorrt.tensorrt.IPoolingLayer
Add a multi-dimension pooling layer to the network. See IPoolingLayer for more information.
Parameters
• input - The input tensor to the layer.
• type - The type of pooling to apply.
• window_size - The size of the pooling window.
Returns The new pooling layer, or None if it could not be created.
add_quantize(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, scale:
tensorrt.tensorrt.ITensor) → tensorrt.tensorrt.IQuantizeLayer
Add a quantization layer to the network. See IQuantizeLayer for more information.
Parameters
• input - A tensor to quantize.
• scale - A tensor with the scale coefficients.
Returns The new quantization layer, or None if it could not be created.
add_ragged_softmax(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, bounds:
tensorrt.tensorrt.ITensor) → tensorrt.tensorrt.IRaggedSoftMaxLayer
Add a ragged softmax layer to the network. See IRaggedSoftMaxLayer for more information.
Parameters
• input - The ZxS input tensor.
• bounds - The Zx1 bounds tensor.
Returns The new ragged softmax layer, or None if it could not be created.
add_reduce(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, op:
tensorrt.tensorrt.ReduceOperation, axes: int, keep_dims: bool) →
tensorrt.tensorrt.IReduceLayer
Add a reduce layer to the network. See IReduceLayer for more information.
Parameters
• input - The input tensor to the layer.
• op - The reduction operation to perform.
• axes - The reduction dimensions. The bit in position i of bitmask axes corresponds to
explicit dimension i of the result. E.g., the least significant bit corresponds to the first
explicit dimension and the next to least significant bit corresponds to the second explicit
dimension.
• keep_dims - The boolean that specifies whether or not to keep the reduced dimensions in
the output of the layer.
Returns The new reduce layer, or None if it could not be created.
add_resize(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.IResizeLayer
Add a resize layer. See IResizeLayer for more information.
Parameters input - The input tensor to the layer.
5.1. INetworkDefinition
57
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Returns The new resize layer, or None if it could not be created.
add_reverse_sequence(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor,
sequence_lens: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.IReverseSequenceLayer
Adds a ReverseSequence layer to the network. See IReverseSequenceLayer for more information.
Parameters
• input - The input tensor to the layer.
• sequence_lens - 1D tensor specifying lengths of sequences to reverse in a batch. The
length of sequence_lens must be equal to the size of the dimension in input specified
by batch_axis.
Returns the new ReverseSequence layer, or None if it could not be created.
add_rnn_v2(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, layer_count: int,
hidden_size: int, max_seq_length: int, op: tensorrt.tensorrt.RNNOperation) →
tensorrt.tensorrt.IRNNv2Layer
Add an RNNv2 layer to the network. See IRNNv2Layer for more information.
Add an layer_count deep RNN layer to the network with hidden_size internal states that can take a
batch with fixed or variable sequence lengths.
Parameters
• input - The input tensor to the layer (see below).
• layer_count - The number of layers in the RNN.
• hidden_size - Size of the internal hidden state for each layer.
• max_seq_length - Maximum sequence length for the input.
• op - The type of RNN to execute.
By default, the layer is configured with RNNDirection.UNIDIRECTION and RNNInputMode.LINEAR . To
change these settings, set IRNNv2Layer.direction and IRNNv2Layer.input_mode .
Weights and biases for the added layer should be set using IRNNv2Layer.set_weights_for_gate()
and IRNNv2Layer.set_bias_for_gate() prior to building an engine using this network.
The input tensors must be of the type float32 or float16 . The layout of the weights is row major and
must be the same datatype as the input tensor. weights contain 8 matrices and bias contains 8 vectors.
See IRNNv2Layer.set_weights_for_gate() and IRNNv2Layer.set_bias_for_gate() for details
on the required input format for weights and bias .
The input ITensor should contain zero or more index dimensions {N1, . . . , Np}, followed by two dimen-
sions, defined as follows:
S_max is the maximum allowed sequence length (number of RNN iterations)
E specifies the embedding length (unless RNNInputMode.SKIP is set, in which case it should match
IRNNv2Layer.hidden_size ).
By default, all sequences in the input are assumed to be size max_seq_length . To provide explicit se-
quence lengths for each input sequence in the batch, set IRNNv2Layer.seq_lengths .
The RNN layer outputs up to three tensors.
58
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
The first output tensor is the output of the final RNN layer across all timesteps, with dimensions {N1, . . . ,
Np, S_max, H}:
N1..Np are the index dimensions specified by the input tensor
S_max is the maximum allowed sequence length (number of RNN iterations)
H is an output hidden state (equal to IRNNv2Layer.hidden_size or 2x IRNNv2Layer.hidden_size )
The second tensor is the final hidden state of the RNN across all layers, and if the RNN is an LSTM (i.e.
IRNNv2Layer.op is RNNOperation.LSTM ), then the third tensor is the final cell state of the RNN across
all layers. Both the second and third output tensors have dimensions {N1, . . . , Np, L, H}:
N1..Np are the index dimensions specified by the input tensor
L is the number of layers in the RNN, equal to IRNNv2Layer.num_layers
H is the hidden state for each layer, equal to IRNNv2Layer.hidden_size if getDirection is
RNNDirection.UNIDIRECTION, and 2x IRNNv2Layer.hidden_size otherwise.
Returns The new RNNv2 layer, or None if it could not be created.
add_scale(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, mode:
tensorrt.tensorrt.ScaleMode, shift: tensorrt.tensorrt.Weights = None, scale:
tensorrt.tensorrt.Weights = None, power: tensorrt.tensorrt.Weights = None) →
tensorrt.tensorrt.IScaleLayer
Add a scale layer to the network. See IScaleLayer for more information.
Parameters
• input - The input tensor to the layer. This tensor is required to have a minimum of 3
dimensions.
• mode - The scaling mode.
• shift - The shift value.
• scale - The scale value.
• power - The power value.
If the weights are available, then the size of weights are dependent on the ScaleMode. For UNIFORM, the
number of weights is equal to 1. For CHANNEL, the number of weights is equal to the channel dimension.
For ELEMENTWISE, the number of weights is equal to the volume of the input.
Returns The new scale layer, or None if it could not be created.
add_scale_nd(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, mode:
tensorrt.tensorrt.ScaleMode, shift: tensorrt.tensorrt.Weights = None, scale:
tensorrt.tensorrt.Weights = None, power: tensorrt.tensorrt.Weights = None, channel_axis:
int) → tensorrt.tensorrt.IScaleLayer
Add a multi-dimension scale layer to the network. See IScaleLayer for more information.
Parameters
• input - The input tensor to the layer. This tensor is required to have a minimum of 3
dimensions.
• mode - The scaling mode.
5.1. INetworkDefinition
59
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• shift - The shift value.
• scale - The scale value.
• power - The power value.
• channel_axis - The channel dimension axis.
If the weights are available, then the size of weights are dependent on the ScaleMode. For UNIFORM, the
number of weights is equal to 1. For CHANNEL, the number of weights is equal to the channel dimension.
For ELEMENTWISE, the number of weights is equal to the volume of the input.
Returns The new scale layer, or None if it could not be created.
add_scatter(self: tensorrt.tensorrt.INetworkDefinition, data: tensorrt.tensorrt.ITensor, indices:
tensorrt.tensorrt.ITensor, updates: tensorrt.tensorrt.ITensor, mode:
tensorrt.tensorrt.ScatterMode) → tensorrt.tensorrt.IScatterLayer
Add a scatter layer to the network. See IScatterLayer for more information.
Parameters
• data - The tensor to get default values from.
• indices - The tensor to get indices from to populate the output tensor.
• updates - The tensor to get values from to populate the output tensor.
• mode - operation mode see IScatterLayer for more info
Returns The new Scatter layer, or None if it could not be created.
add_select(self: tensorrt.tensorrt.INetworkDefinition, condition: tensorrt.tensorrt.ITensor, then_input:
tensorrt.tensorrt.ITensor, else_input: tensorrt.tensorrt.ITensor) → tensorrt.tensorrt.ISelectLayer
Add a select layer. See ISelectLayer for more information.
Parameters
• condition - The condition tensor to the layer.
• then_input - The then input tensor to the layer.
• else_input - The else input tensor to the layer.
Returns The new select layer, or None if it could not be created.
add_shape(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.IShapeLayer
Add a shape layer to the network. See IShapeLayer for more information.
Parameters input - The input tensor to the layer.
Returns The new shape layer, or None if it could not be created.
add_shuffle(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.IShuffleLayer
Add a shuffle layer to the network. See IShuffleLayer for more information.
Parameters input - The input tensor to the layer.
Returns The new shuffle layer, or None if it could not be created.
add_slice(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, start:
tensorrt.tensorrt.Dims, shape: tensorrt.tensorrt.Dims, stride: tensorrt.tensorrt.Dims) →
tensorrt.tensorrt.ISliceLayer
Add a slice layer to the network. See ISliceLayer for more information.
Parameters
60
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• input - The input tensor to the layer.
• start - The start offset.
• shape - The output shape.
• stride - The slicing stride. Positive, negative, zero stride values, and combinations of
them in different dimensions are allowed.
Returns The new slice layer, or None if it could not be created.
add_softmax(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.ISoftMaxLayer
Add a softmax layer to the network. See ISoftMaxLayer for more information.
Parameters input - The input tensor to the layer.
Returns The new softmax layer, or None if it could not be created.
add_topk(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, op:
tensorrt.tensorrt.TopKOperation, k: int, axes: int) → tensorrt.tensorrt.ITopKLayer
Add a TopK layer to the network. See ITopKLayer for more information.
The TopK layer has two outputs of the same dimensions. The first contains data values, the second contains
index positions for the values. Output values are sorted, largest first for operation TopKOperation.MAX
and smallest first for operation TopKOperation.MIN .
Currently only values of K up to 3840 are supported.
Parameters
• input - The input tensor to the layer.
• op - Operation to perform.
• k - Number of elements to keep.
• axes - The reduction dimensions. The bit in position i of bitmask axes corresponds to
explicit dimension i of the result. E.g., the least significant bit corresponds to the first
explicit dimension and the next to least significant bit corresponds to the second explicit
dimension. Currently axes must specify exactly one dimension, and it must be one of the
last four dimensions.
Returns The new TopK layer, or None if it could not be created.
add_unary(self: tensorrt.tensorrt.INetworkDefinition, input: tensorrt.tensorrt.ITensor, op:
tensorrt.tensorrt.UnaryOperation) → tensorrt.tensorrt.IUnaryLayer
Add a unary layer to the network. See IUnaryLayer for more information.
Parameters
• input - The input tensor to the layer.
• op - The operation to apply.
Returns The new unary layer, or None if it could not be created.
property builder
The builder from which this INetworkDefinition was created.
See IBuilder for more information.
get_input(self: tensorrt.tensorrt.INetworkDefinition, index: int) → tensorrt.tensorrt.ITensor
Get the input tensor specified by the given index.
Parameters index - The index of the input tensor.
5.1. INetworkDefinition
61
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Returns The tensor, or None if it is out of range.
get_layer(self: tensorrt.tensorrt.INetworkDefinition, index: int) → tensorrt.tensorrt.ILayer
Get the layer specified by the given index.
Parameters index - The index of the layer.
Returns The layer, or None if it is out of range.
get_output(self: tensorrt.tensorrt.INetworkDefinition, index: int) → tensorrt.tensorrt.ITensor
Get the output tensor specified by the given index.
Parameters index - The index of the output tensor.
Returns The tensor, or None if it is out of range.
mark_output(self: tensorrt.tensorrt.INetworkDefinition, tensor: tensorrt.tensorrt.ITensor) → None
Mark a tensor as an output.
Parameters tensor - The tensor to mark.
mark_output_for_shapes(self: tensorrt.tensorrt.INetworkDefinition, tensor: tensorrt.tensorrt.ITensor) →
bool
Enable tensor’s value to be computed by IExecutionContext.get_shape_binding().
Parameters tensor - The tensor to unmark as an output tensor. The tensor must be of type
int32 and have no more than one dimension.
Returns True if successful, False if tensor is already marked as an output.
remove_tensor(self: tensorrt.tensorrt.INetworkDefinition, tensor: tensorrt.tensorrt.ITensor) → None
Remove a tensor from the network.
Parameters tensor - The tensor to remove
It is illegal to remove a tensor that is the input or output of a layer. if this method is called with such a
tensor, a warning will be emitted on the log and the call will be ignored.
set_weights_name(self: tensorrt.tensorrt.INetworkDefinition, weights: tensorrt.tensorrt.Weights, name:
str) → bool
Associate a name with all current uses of the given weights.
The name must be set after the Weights are used in the network. Lookup is associative. The name applies
to all Weights with matching type, value pointer, and count. If Weights with a matching value pointer, but
different type or count exists in the network, an error message is issued, the name is rejected, and return
false. If the name has already been used for other weights, return false. None causes the weights to become
unnamed, i.e. clears any previous name.
Parameters
• weights - The weights to be named.
• name - The name to associate with the weights.
Returns true on success.
unmark_output(self: tensorrt.tensorrt.INetworkDefinition, tensor: tensorrt.tensorrt.ITensor) → None
Unmark a tensor as a network output.
Parameters tensor - The tensor to unmark as an output tensor.
unmark_output_for_shapes(self: tensorrt.tensorrt.INetworkDefinition, tensor: tensorrt.tensorrt.ITensor)
→ bool
Undo mark_output_for_shapes() .
Parameters tensor - The tensor to unmark as an output tensor.
62
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Returns True if successful, False if tensor is not marked as an output.
5.2 Layer Base Classes
5.2.1 ITensor
tensorrt.TensorLocation
The physical location of the data.
Members:
DEVICE : Data is stored on the device.
HOST : Data is stored on the device.
tensorrt.TensorFormat
Format of the input/output tensors.
This enum is used by both plugins and network I/O tensors.
For more information about data formats, see the topic “Data Format Description” located in the Ten-
sorRT Developer Guide (https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.
html).
Members:
LINEAR : Row major linear format.
For a tensor with dimensions {N, C, H, W}, the W axis always has unit stride, and the stride of
every other axis is at least the product of the next dimension times the next stride. the strides are
the same as for a C array with dimensions [N][C][H][W].
CHW2 : Two wide channel vectorized row major format.
This format is bound to FP16. It is only available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with
dimensions [N][(C+1)/2][H][W][2], with the tensor coordinates (n, c, h, w) mapping to array
subscript [n][c/2][h][w][c%2].
HWC8 : Eight channel format where C is padded to a multiple of 8.
This format is bound to FP16. It is only available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to the array with
dimensions [N][H][W][(C+7)/8*8], with the tensor coordinates (n, c, h, w) mapping to array
subscript [n][h][w][c].
CHW4 : Four wide channel vectorized row major format. This format is bound to INT8. It is only
available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with
dimensions [N][(C+3)/4][H][W][4], with the tensor coordinates (n, c, h, w) mapping to array
subscript [n][c/4][h][w][c%4].
CHW16 : Sixteen wide channel vectorized row major format.
This format is bound to FP16. It is only available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with
dimensions [N][(C+15)/16][H][W][16], with the tensor coordinates (n, c, h, w) mapping to array
subscript [n][c/16][h][w][c%16].
5.2. Layer Base Classes
63
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
CHW32 : Thirty-two wide channel vectorized row major format.
This format is only available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with
dimensions [N][(C+31)/32][H][W][32], with the tensor coordinates (n, c, h, w) mapping to array
subscript [n][c/32][h][w][c%32].
DHWC8 : Eight channel format where C is padded to a multiple of 8.
This format is bound to FP16, and it is only available for dimensions >= 4.
For a tensor with dimensions {N, C, D, H, W}, the memory layout is equivalent to an array with
dimensions [N][D][H][W][(C+7)/8*8], with the tensor coordinates (n, c, d, h, w) mapping to
array subscript [n][d][h][w][c].
CDHW32 : Thirty-two wide channel vectorized row major format with 3 spatial dimensions.
This format is bound to FP16 and INT8. It is only available for dimensions >= 4.
For a tensor with dimensions {N, C, D, H, W}, the memory layout is equivalent to a C array with
dimensions [N][(C+31)/32][D][H][W][32], with the tensor coordinates (n, d, c, h, w) mapping
to array subscript [n][c/32][d][h][w][c%32].
HWC : Non-vectorized channel-last format. This format is bound to FP32 and is only available for
dimensions >= 3.
DLA_LINEAR : DLA planar format. Row major format. The stride for stepping along the H axis
is rounded up to 64 bytes.
This format is bound to FP16/Int8 and is only available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with
dimensions [N][C][H][roundUp(W, 64/elementSize)] where elementSize is 2 for FP16 and 1 for
Int8, with the tensor coordinates (n, c, h, w) mapping to array subscript [n][c][h][w].
DLA_HWC4 : DLA image format. channel-last format. C can only be 1, 3, 4. If C == 3 it will be
rounded to 4. The stride for stepping along the H axis is rounded up to 32 bytes.
This format is bound to FP16/Int8 and is only available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, with C’ is 1, 4, 4 when C is 1, 3, 4 respec-
tively, the memory layout is equivalent to a C array with dimensions [N][H][roundUp(W,
32/C’/elementSize)][C’] where elementSize is 2 for FP16 and 1 for Int8, C’ is the rounded C.
The tensor coordinates (n, c, h, w) maps to array subscript [n][h][w][c].
HWC16 : Sixteen channel format where C is padded to a multiple of 16. This format is bound to
FP16. It is only available for dimensions >= 3.
For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to the array with
dimensions [N][H][W][(C+15)/16*16], with the tensor coordinates (n, c, h, w) mapping to array
subscript [n][h][w][c].
DHWC : Non-vectorized channel-last format. This format is bound to FP32. It is only available for
dimensions >= 4.
class tensorrt.ITensor
A tensor in an INetworkDefinition .
Variables
• name - str The tensor name. For a network input, the name is assigned by the application.
For tensors which are layer outputs, a default name is assigned consisting of the layer name
followed by the index of the output in brackets.
64
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• shape - Dims The shape of a tensor. For a network input the shape is assigned by the
application. For a network output it is computed based on the layer parameters and the
inputs to the layer. If a tensor size or a parameter is modified in the network, the shape of
all dependent tensors will be recomputed. This call is only legal for network input tensors,
since the shape of layer output tensors are inferred based on layer inputs and parameters.
• dtype - DataType The data type of a tensor. The type is unchanged if the type is invalid
for the given tensor.
• broadcast_across_batch - bool Whether to enable broadcast of tensor across the batch.
When a tensor is broadcast across a batch, it has the same value for every member in the batch.
Memory is only allocated once for the single member. This method is only valid for network
input tensors, since the flags of layer output tensors are inferred based on layer inputs and
parameters. If this state is modified for a tensor in the network, the states of all dependent
tensors will be recomputed.
• location - TensorLocation The storage location of a tensor.
• is_network_input - bool Whether the tensor is a network input.
• is_network_output - bool Whether the tensor is a network output.
• dynamic_range - Tuple[float, float] A tuple containing the [minimum, maximum]
of the dynamic range, or None if the range was not set.
• is_shape - bool Whether the tensor is a shape tensor.
• allowed_formats - int32 The allowed set of TensorFormat candidates. This should be
an integer consisting of one or more TensorFormat s, combined via bitwise OR after bit
shifting. For example, 1 << int(TensorFormat.CHW4) | 1 << int(TensorFormat.
CHW32).
get_dimension_name(self: tensorrt.tensorrt.ITensor, index: int) → str
Get the name of an input dimension.
Parameters index - index of the dimension.
Returns name of the dimension, or null if dimension is unnamed.
reset_dynamic_range(self: tensorrt.tensorrt.ITensor) → None
Undo the effect of setting the dynamic range.
set_dimension_name(self: tensorrt.tensorrt.ITensor, index: int, name: str) → None
Name a dimension of an input tensor.
Associate a runtime dimension of an input tensor with a symbolic name. Dimensions with the same non-
empty name must be equal at runtime. Knowing this equality for runtime dimensions may help the Ten-
sorRT optimizer. Both runtime and build-time dimensions can be named. If the function is called again,
with the same index, it will overwrite the previous name. If None is passed as name, it will clear the name
of the dimension.
For example, setDimensionName(0, “n”) associates the symbolic name “n” with the leading dimension.
Parameters
• index - index of the dimension.
• name - name of the dimension.
set_dynamic_range(self: tensorrt.tensorrt.ITensor, min: float, max: float) → bool
Set dynamic range for the tensor. NOTE: It is suggested to use tensor.dynamic_range = (min, max)
instead.
Parameters
5.2. Layer Base Classes
65
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• min - Minimum of the dynamic range.
• max - Maximum of the dyanmic range.
Returns true if succeed in setting range. Otherwise false.
5.2.2 ILayer
tensorrt.LayerType
Type of Layer
Members:
CONVOLUTION : Convolution layer
FULLY_CONNECTED : Fully connected layer
GRID_SAMPLE : Grid sample layer
NMS : NMS layer
ACTIVATION : Activation layer
POOLING : Pooling layer
LRN : LRN layer
SCALE : Scale layer
SOFTMAX : Softmax layer
DECONVOLUTION : Deconvolution layer
CONCATENATION : Concatenation layer
ELEMENTWISE : Elementwise layer
PLUGIN : Plugin layer
UNARY : Unary layer
PADDING : Padding layer
SHUFFLE : Shuffle layer
REDUCE : Reduce layer
TOPK : TopK layer
GATHER : Gather layer
MATRIX_MULTIPLY : Matrix multiply layer
RAGGED_SOFTMAX : Ragged softmax layer
CONSTANT : Constant layer
RNN_V2 : RNNv2 layer
IDENTITY : Identity layer
CAST : Cast layer
PLUGIN_V2 : PluginV2 layer
SLICE : Slice layer
SHAPE : Shape layer
66
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
PARAMETRIC_RELU : Parametric ReLU layer
RESIZE : Resize layer
TRIP_LIMIT : Loop Trip limit layer
RECURRENCE : Loop Recurrence layer
ITERATOR : Loop Iterator layer
LOOP_OUTPUT : Loop output layer
SELECT : Select layer
ASSERTION : Assertion layer
FILL : Fill layer
QUANTIZE : Quantize layer
DEQUANTIZE : Dequantize layer
CONDITION : If-conditional Condition layer
CONDITIONAL_INPUT : If-conditional input layer
CONDITIONAL_OUTPUT : If-conditional output layer
SCATTER : Scatter layer
EINSUM : Einsum layer
ONE_HOT : OneHot layer
NON_ZERO : NonZero layer
REVERSE_SEQUENCE : ReverseSequence layer
NORMALIZATION : Normalization layer
class tensorrt.ILayer
Base class for all layer classes in an INetworkDefinition .
Variables
• name - str The name of the layer.
• type - LayerType The type of the layer.
• num_inputs - int The number of inputs of the layer.
• num_outputs - int The number of outputs of the layer.
• precision - DataType The computation precision.
• precision_is_set - bool Whether the precision is set or not.
Ival metadata str The per-layer metadata.
get_input(self: tensorrt.tensorrt.ILayer, index: int) → tensorrt.tensorrt.ITensor
Get the layer input corresponding to the given index.
Parameters index - The index of the input tensor.
Returns The input tensor, or None if the index is out of range.
get_output(self: tensorrt.tensorrt.ILayer, index: int) → tensorrt.tensorrt.ITensor
Get the layer output corresponding to the given index.
Parameters index - The index of the output tensor.
5.2. Layer Base Classes
67
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Returns The output tensor, or None if the index is out of range.
get_output_type(self: tensorrt.tensorrt.ILayer, index: int) → tensorrt.tensorrt.DataType
Get the output type of the layer.
Parameters index - The index of the output tensor.
Returns The output precision. Default : DataType.FLOAT.
output_type_is_set(self: tensorrt.tensorrt.ILayer, index: int) → bool
Whether the output type has been set for this layer.
Parameters index - The index of the output.
Returns Whether the output type has been explicitly set.
reset_output_type(self: tensorrt.tensorrt.ILayer, index: int) → None
Reset output type of this layer.
Parameters index - The index of the output.
reset_precision(self: tensorrt.tensorrt.ILayer) → None
Reset the computation precision of the layer.
set_input(self: tensorrt.tensorrt.ILayer, index: int, tensor: tensorrt.tensorrt.ITensor) → None
Set the layer input corresponding to the given index.
Parameters
• index - The index of the input tensor.
• tensor - The input tensor.
set_output_type(self: tensorrt.tensorrt.ILayer, index: int, dtype: tensorrt.tensorrt.DataType) → None
Constraint layer to generate output data with given type. Note that this method cannot be used to set the
data type of the second output tensor of the topK layer. The data type of the second output tensor of the
topK layer is always int32 .
Parameters
• index - The index of the output tensor to set the type.
• dtype - DataType of the output.
5.3 Layers
5.3.1 PaddingMode
tensorrt.PaddingMode
Enumerates types of padding available in convolution, deconvolution and pooling layers. Padding mode
takes precedence if both padding_mode and pre_padding are set.
EXPLICIT* corresponds to explicit padding.
SAME* implicitly calculates padding such that the output dimensions are the same as the input
dimensions. For convolution and pooling, output dimensions are determined by ceil(input dimensions,
stride).
CAFFE* corresponds to symmetric padding.
68
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Members:
EXPLICIT_ROUND_DOWN : Use explicit padding, rounding the output size down
EXPLICIT_ROUND_UP : Use explicit padding, rounding the output size up
SAME_UPPER : Use SAME padding, with pre_padding <= post_padding
SAME_LOWER : Use SAME padding, with pre_padding >= post_padding
CAFFE_ROUND_DOWN : Use CAFFE padding, rounding the output size down
CAFFE_ROUND_UP : Use CAFFE padding, rounding the output size up
5.3.2 IConvolutionLayer
class tensorrt.IConvolutionLayer
A convolution layer in an INetworkDefinition .
This layer performs a correlation operation between 3-dimensional filter with a 4-dimensional tensor to produce
another 4-dimensional tensor.
An optional bias argument is supported, which adds a per-channel constant to each value in the output.
Variables
• kernel_size - DimsHW The HW kernel size of the convolution.
• num_output_maps - int The number of output maps for the convolution.
• stride - DimsHW The stride of the convolution. Default: (1, 1)
• padding - DimsHW The padding of the convolution. The input will be zero-padded by this
number of elements in the height and width directions. If the padding is asymmetric, this
value corresponds to the pre-padding. Default: (0, 0)
• pre_padding - DimsHW The pre-padding. The start of input will be zero-padded by this
number of elements in the height and width directions. Default: (0, 0)
• post_padding - DimsHW The post-padding. The end of input will be zero-padded by this
number of elements in the height and width directions. Default: (0, 0)
• padding_mode - PaddingMode The padding mode. Padding mode takes precedence if both
IConvolutionLayer.padding_mode and either IConvolutionLayer.pre_padding or
IConvolutionLayer.post_padding are set.
• num_groups - int The number of groups for a convolution. The input tensor channels are
divided into this many groups, and a convolution is executed for each group, using a filter
per group. The results of the group convolutions are concatenated to form the output. Note
When using groups in int8 mode, the size of the groups (i.e. the channel count divided by
the group count) must be a multiple of 4 for both input and output. Default: 1.
• kernel - Weights The kernel weights for the convolution. The weights are specified as a
contiguous array in GKCRS order, where G is the number of groups, K the number of output
feature maps, C the number of input channels, and R and S are the height and width of the
filter.
• bias - Weights The bias weights for the convolution. Bias is optional. To omit bias, set
this to an empty Weights object. The bias is applied per-channel, so the number of weights
(if non-zero) must be equal to the number of output feature maps.
• dilation - DimsHW The dilation for a convolution. Default: (1, 1)
5.3. Layers
69
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• kernel_size_nd - Dims The multi-dimension kernel size of the convolution.
• stride_nd - Dims The multi-dimension stride of the convolution. Default: (1, . . . , 1)
• padding_nd - Dims The multi-dimension padding of the convolution. The input will be
zero-padded by this number of elements in each dimension. If the padding is asymmetric,
this value corresponds to the pre-padding. Default: (0, . . . , 0)
• dilation_nd - Dims The multi-dimension dilation for the convolution. Default: (1, . . . , 1)
5.3.3 IFullyConnectedLayer
class tensorrt.IFullyConnectedLayer
A fully connected layer in an INetworkDefinition .
This layer expects an input tensor of three or more non-batch dimensions. The input is automatically reshaped
into an MxV tensor X, where V is a product of the last three dimensions and M is a product of the remaining
dimensions (where the product over 0 dimensions is defined as 1). For example:
• If the input tensor has shape {C, H, W}, then the tensor is reshaped into {1, C*H*W} .
• If the input tensor has shape {P, C, H, W}, then the tensor is reshaped into {P, C*H*W} .
The layer then performs:
Y := matmul(X, WT ) + bias
Where X is the MxV tensor defined above, W is the KxV weight tensor of the layer, and bias is a row vector size K
that is broadcasted to MxK . K is the number of output channels, and configurable via IFullyConnectedLayer.
num_output_channels . If bias is not specified, it is implicitly 0 .
The MxK result Y is then reshaped such that the last three dimensions are {K, 1, 1} and the remaining dimensions
match the dimensions of the input tensor. For example:
• If the input tensor has shape {C, H, W}, then the output tensor will have shape {K, 1, 1} .
• If the input tensor has shape {P, C, H, W}, then the output tensor will have shape {P, K, 1, 1} .
Variables
• num_output_channels - int The number of output channels K from the fully connected
layer.
• kernel - Weights The kernel weights, given as a KxC matrix in row-major order.
• bias - Weights The bias weights. Bias is optional. To omit bias, set this to an empty
Weights object.
5.3.4 IGridSampleLayer
tensorrt.InterpolationMode
Various modes of interpolation, used in resize and grid_sample layers.
Members:
NEAREST : 1D, 2D, and 3D nearest neighbor interpolation.
LINEAR : Supports linear, bilinear, trilinear interpolation.
CUBIC : Supports bicubic interpolation.
70
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
tensorrt.SampleMode
Controls how ISliceLayer and IGridSample handles out of bounds coordinates
Members:
STRICT_BOUNDS : Fail with error when the coordinates are out of bounds.
DEFAULT : [DEPRECATED] Use STRICT_BOUNDS.
WRAP : Coordinates wrap around periodically.
CLAMP : Out of bounds indices are clamped to bounds
FILL : Use fill input value when coordinates are out of bounds.
REFLECT : Coordinates reflect.
class tensorrt.IGridSampleLayer
A grid sample layer in an INetworkDefinition .
This layer uses an input tensor and a grid tensor to produce an interpolated output tensor. The input and
grid tensors must shape tensors of rank 4. The only supported SampleMode s are trt.samplemode.CLAMP,
trt.samplemode.FILL, and trt.samplemode.REFLECT.
Variables
• interpolation_mode - class:InterpolationMode The interpolation type to use. Defaults
to LINEAR.
• align_corners - class:bool the align mode to use. Defaults to False.
• sample_mode - SampleMode The sample mode to use. Defaults to FILL.
5.3.5 IActivationLayer
tensorrt.ActivationType
The type of activation to perform.
Members:
RELU : Rectified Linear activation
SIGMOID : Sigmoid activation
TANH : Hyperbolic Tangent activation
LEAKY_RELU : Leaky Relu activation: f(x) = x if x >= 0, f(x) = alpha * x if x < 0
ELU : Elu activation: f(x) = x if x >= 0, f(x) = alpha * (exp(x) - 1) if x < 0
SELU : Selu activation: f(x) = beta * x if x > 0, f(x) = beta * (alpha * exp(x) - alpha) if x <= 0
SOFTSIGN : Softsign activation: f(x) = x / (1 + abs(x))
SOFTPLUS : Softplus activation: f(x) = alpha * log(exp(beta * x) + 1)
CLIP : Clip activation: f(x) = max(alpha, min(beta, x))
HARD_SIGMOID : Hard sigmoid activation: f(x) = max(0, min(1, alpha * x + beta))
SCALED_TANH : Scaled Tanh activation: f(x) = alpha * tanh(beta * x)
THRESHOLDED_RELU : Thresholded Relu activation: f(x) = x if x > alpha, f(x) = 0 if x <= alpha
5.3. Layers
71
|
||
|
|
|