|
|
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
class tensorrt.IActivationLayer
An Activation layer in an INetworkDefinition . This layer applies a per-element activation function to its
input. The output has the same shape as the input.
Variables
• type - ActivationType The type of activation to be performed.
• alpha - float The alpha parameter that is used by some parametric activations
(LEAKY_RELU, ELU, SELU, SOFTPLUS, CLIP, HARD_SIGMOID, SCALED_TANH).
Other activations ignore this parameter.
• beta - float The beta parameter that is used by some parametric activations (SELU, SOFT-
PLUS, CLIP, HARD_SIGMOID, SCALED_TANH). Other activations ignore this parame-
ter.
5.3.6 IPoolingLayer
tensorrt.PoolingType
The type of pooling to perform in a pooling layer.
Members:
MAX : Maximum over elements
AVERAGE : Average over elements. If the tensor is padded, the count includes the padding
MAX_AVERAGE_BLEND : Blending between the max pooling and average pooling:
(1-
blendFactor)*maxPool + blendFactor*avgPool
class tensorrt.IPoolingLayer
A Pooling layer in an INetworkDefinition . The layer applies a reduction operation within a window over the
input.
Variables
• type - PoolingType The type of pooling to be performed.
• window_size - DimsHW The window size for pooling.
• stride - DimsHW The stride for pooling. Default: (1, 1)
• padding - DimsHW The padding for pooling. Default: (0, 0)
• pre_padding - DimsHW The pre-padding. The start of input will be zero-padded by this
number of elements in the height and width directions. Default: (0, 0)
• post_padding - DimsHW The post-padding. The end of input will be zero-padded by this
number of elements in the height and width directions. Default: (0, 0)
• padding_mode - PaddingMode The padding mode. Padding mode takes precedence
if both IPoolingLayer.padding_mode and either IPoolingLayer.pre_padding or
IPoolingLayer.post_padding are set.
• blend_factor - float The blending factor for the max_average_blend mode:
maxaverageblendP ool = (1 - blendF actor) * maxP ool + blendF actor * avgP ool .
blend_factor is a user value in [0,1] with the default value of 0.0. This value only applies
for the PoolingType.MAX_AVERAGE_BLEND mode.
• average_count_excludes_padding - bool Whether average pooling uses as a denom-
inator the overlap area between the window and the unpadded input. If this is not set, the
denominator is the overlap between the pooling window and the padded input. Default: True
72
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• window_size_nd - Dims The multi-dimension window size for pooling.
• stride_nd - Dims The multi-dimension stride for pooling. Default: (1, . . . , 1)
• padding_nd - Dims The multi-dimension padding for pooling. Default: (0, . . . , 0)
5.3.7 ILRNLayer
class tensorrt.ILRNLayer
A LRN layer in an INetworkDefinition . The output size is the same as the input size.
Variables
• window_size - int The LRN window size. The window size must be odd and in the range
of [1, 15].
• alpha - float The LRN alpha value. The valid range is [-1e20, 1e20].
• beta - float The LRN beta value. The valid range is [0.01, 1e5f].
• k - float The LRN K value. The valid range is [1e-5, 1e10].
5.3.8 IScaleLayer
tensorrt.ScaleMode
Controls how scale is applied in a Scale layer.
Members:
UNIFORM : Identical coefficients across all elements of the tensor.
CHANNEL : Per-channel coefficients. The channel dimension is assumed to be the third to last di-
mension.
ELEMENTWISE : Elementwise coefficients.
class tensorrt.IScaleLayer
A Scale layer in an INetworkDefinition .
This layer applies a per-element computation to its input:
output = (input * scale + sℎift)power
The coefficients can be applied on a per-tensor, per-channel, or per-element basis.
Note If the number of weights is 0, then a default value is used for shift, power, and scale. The default shift is 0,
the default power is 1, and the default scale is 1.
The output size is the same as the input size.
Note The input tensor for this layer is required to have a minimum of 3 dimensions.
Variables
• mode - ScaleMode The scale mode.
• shift - Weights The shift value.
• scale - Weights The scale value.
• power - Weights The power value.
• channel_axis - int The channel axis.
5.3. Layers
73
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.9 ISoftMaxLayer
class tensorrt.ISoftMaxLayer
A Softmax layer in an INetworkDefinition .
This layer applies a per-channel softmax to its input.
The output size is the same as the input size.
Variables axes - int The axis along which softmax is computed. Currently, only one axis can be
set.
The axis is specified by setting the bit corresponding to the axis to 1, as a bit mask.
For example, consider an NCHW tensor as input (three non-batch dimensions).
In implicit mode :
Bit 0 corresponds to the C dimension boolean.
Bit 1 corresponds to the H dimension boolean.
Bit 2 corresponds to the W dimension boolean.
By default, softmax is performed on the axis which is the number of axes minus three. It is 0 if there are fewer
than 3 non-batch axes. For example, if the input is NCHW, the default axis is C. If the input is NHW, then the
default axis is H.
In explicit mode :
Bit 0 corresponds to the N dimension boolean.
Bit 1 corresponds to the C dimension boolean.
Bit 2 corresponds to the H dimension boolean.
Bit 3 corresponds to the W dimension boolean.
By default, softmax is performed on the axis which is the number of axes minus three. It is 0 if
there are fewer than 3 axes. For example, if the input is NCHW, the default axis is C. If the input
is NHW, then the default axis is N.
For example, to perform softmax on axis R of a NPQRCHW input, set bit 2 with implicit batch mode,
set bit 3 with explicit batch mode.
On Xavier, this layer is not supported on DLA. Otherwise, the following constraints must be satisfied to execute
this layer on DLA:
• Axis must be one of the channel or spatial dimensions.
• There are two classes of supported input sizes:
- Non-axis, non-batch dimensions are all 1 and the axis dimension is at most 8192. This is the recom-
mended case for using softmax since it is the most accurate.
- At least one non-axis, non-batch dimension greater than 1 and the axis dimension is at most 1024.
Note that in this case, there may be some approximation error as the axis dimension size approaches
the upper bound. See the TensorRT Developer Guide for more details on the approximation error.
74
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.10 IConcatenationLayer
class tensorrt.IConcatenationLayer
A concatenation layer in an INetworkDefinition .
The output channel size is the sum of the channel sizes of the inputs. The other output sizes are the same as the
other input sizes, which must all match.
Variables axis - int The axis along which concatenation occurs. The default axis is the number
of tensor dimensions minus three, or zero if the tensor has fewer than three dimensions. For
example, for a tensor with dimensions NCHW, it is C. For implicit batch mode, the number of
tensor dimensions does NOT include the implicit batch dimension.
5.3.11 IDeconvolutionLayer
class tensorrt.IDeconvolutionLayer
A deconvolution layer in an INetworkDefinition .
Variables
•
kernel_size - DimsHW The HW kernel size of the convolution.
•
num_output_maps - int The number of output feature maps for the deconvolution.
•
stride - DimsHW The stride of the deconvolution. Default: (1, 1)
•
padding - DimsHW The padding of the deconvolution. The input will be zero-padded by
this number of elements in the height and width directions. Padding is symmetric. Default:
(0, 0)
•
pre_padding - DimsHW The pre-padding. The start of input will be zero-padded by this
number of elements in the height and width directions. Default: (0, 0)
•
post_padding - DimsHW The post-padding. The end of input will be zero-padded by this
number of elements in the height and width directions. Default: (0, 0)
•
padding_mode - PaddingMode The padding mode. Padding mode takes precedence
if both IDeconvolutionLayer.padding_mode and either IDeconvolutionLayer.
pre_padding or IDeconvolutionLayer.post_padding are set.
•
num_groups - int The number of groups for a deconvolution. The input tensor channels
are divided into this many groups, and a deconvolution is executed for each group, using a
filter per group. The results of the group convolutions are concatenated to form the output.
Note When using groups in int8 mode, the size of the groups (i.e. the channel count divided
by the group count) must be a multiple of 4 for both input and output. Default: 1
•
kernel - Weights The kernel weights for the deconvolution. The weights are specified as
a contiguous array in CKRS order, where C the number of input channels, K the number of
output feature maps, and R and S are the height and width of the filter.
•
bias - Weights The bias weights for the deconvolution. Bias is optional. To omit bias,
set this to an empty Weights object. The bias is applied per-feature-map, so the number of
weights (if non-zero) must be equal to the number of output feature maps.
•
kernel_size_nd - Dims The multi-dimension kernel size of the convolution.
•
stride_nd - Dims The multi-dimension stride of the deconvolution. Default: (1, . . . , 1)
•
padding_nd - Dims The multi-dimension padding of the deconvolution. The input will be
zero-padded by this number of elements in each dimension. Padding is symmetric. Default:
(0, . . . , 0)
5.3. Layers
75
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.12 IElementWiseLayer
tensorrt.ElementWiseOperation
The binary operations that may be performed by an ElementWise layer.
Members:
SUM : Sum of the two elements
PROD : Product of the two elements
MAX : Max of the two elements
MIN : Min of the two elements
SUB : Subtract the second element from the first
DIV : Divide the first element by the second
POW : The first element to the power of the second element
FLOOR_DIV : Floor division of the first element by the second
AND : Logical AND of two elements
OR : Logical OR of two elements
XOR : Logical XOR of two elements
EQUAL : Check if two elements are equal
GREATER : Check if element in first tensor is greater than corresponding element in second tensor
LESS : Check if element in first tensor is less than corresponding element in second tensor
class tensorrt.IElementWiseLayer
A elementwise layer in an INetworkDefinition .
This layer applies a per-element binary operation between corresponding elements of two tensors.
The input dimensions of the two input tensors must be equal, and the output tensor is the same size as each input.
Variables op - ElementWiseOperation The binary operation for the layer.
5.3.13 IGatherLayer
class tensorrt.IGatherLayer
A gather layer in an INetworkDefinition .
Variables
• axis - int The non-batch dimension axis to gather on. The axis must be less than the
number of non-batch dimensions in the data input.
• num_elementwise_dims - int The number of leading dimensions of indices tensor to
be handled elementwise. For GatherMode.DEFAULT, it must be 0 if there is an im-
plicit batch dimension. It can be 0 or 1 if there is not an implicit batch dimension.
For GatherMode::kND, it can be between 0 and one less than rank(data). For Gather-
Mode::kELEMENT, it must be 0.
• mode - GatherMode The gather mode.
76
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.14 RNN Layers
tensorrt.RNNOperation
The RNN operations that may be performed by an RNN layer.
Equation definitions
In the equations below, we use the following naming convention:
t := current time step
i := input gate
o := output gate
f := forget gate
z := update gate
r := reset gate
c := cell gate
h := hidden gate
g[t] denotes the output of gate g at timestep t, e.g.`f[t]` is the output of the forget gate f .
X[t] := input tensor for timestep t
C[t] := cell state for timestep t
H[t] := hidden state for timestep t
W[g] := W (input) parameter weight matrix for gate g
R[g] := U (recurrent) parameter weight matrix for gate g
Wb[g] := W (input) parameter bias vector for gate g
Rb[g] := U (recurrent) parameter bias vector for gate g
Unless otherwise specified, all operations apply pointwise to elements of each operand tensor.
ReLU(X) := max(X, 0)
tanh(X) := hyperbolic tangent of X
sigmoid(X) := 1 / (1 + exp(-X))
exp(X) := e^X
A.B denotes matrix multiplication of A and B .
A*B denotes pointwise multiplication of A and B .
Equations
Depending on the value of RNNOperation chosen, each sub-layer of the RNN layer will perform one
of the following operations:
RELU
H[t] := ReLU(W [i].X[t] + R[i].H[t - 1] + W b[i] + Rb[i])
TANH
5.3. Layers
77
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
H[t] := tanℎ(W [i].X[t] + R[i].H[t - 1] + W b[i] + Rb[i])
LSTM
i[t] := sigmoid(W [i].X[t] + R[i].H[t - 1] + W b[i] + Rb[i])
f [t] := sigmoid(W [f].X[t] + R[f].H[t - 1] + W b[f] + Rb[f])
o[t] := sigmoid(W [o].X[t] + R[o].H[t - 1] + W b[o] + Rb[o])
c[t] := tanℎ(W [c].X[t] + R[c].H[t - 1] + W b[c] + Rb[c])
C[t] := f[t] * C[t - 1] + i[t] * c[t]
H[t] := o[t] * tanℎ(C[t])
GRU
z[t] := sigmoid(W [z].X[t] + R[z].H[t - 1] + W b[z] + Rb[z])
r[t] := sigmoid(W [r].X[t] + R[r].H[t - 1] + W b[r] + Rb[r])
ℎ[t] := tanℎ(W [ℎ].X[t] + r[t] * (R[ℎ].H[t - 1] + Rb[ℎ]) + W b[ℎ])
H[t] := (1 - z[t]) * ℎ[t] + z[t] * H[t - 1]
Members:
RELU : Single gate RNN w/ ReLU activation
TANH : Single gate RNN w/ TANH activation
LSTM : Four-gate LSTM network w/o peephole connections
GRU : Three-gate network consisting of Gated Recurrent Units
tensorrt.RNNDirection
The RNN direction that may be performed by an RNN layer.
Members:
UNIDIRECTION : Network iterates from first input to last input
BIDIRECTION : Network iterates from first to last (and vice versa) and outputs concatenated
tensorrt.RNNInputMode
The RNN input modes that may occur with an RNN layer.
If the RNN is configured with RNNInputMode.LINEAR , then for each gate g in the first layer of
the RNN, the input vector X[t] (length E) is left-multiplied by the gate’s corresponding weight ma-
trix W[g] (dimensions HxE) as usual, before being used to compute the gate output as described by
RNNOperation .
If the RNN is configured with RNNInputMode.SKIP , then this initial matrix multiplication is
“skipped” and W[g] is conceptually an identity matrix. In this case, the input vector X[t] must have
length H (the size of the hidden state).
Members:
LINEAR : Perform the normal matrix multiplication in the first recurrent layer
SKIP : No operation is performed on the first recurrent layer
78
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.14.1 IRNNv2Layer
tensorrt.RNNGateType
The RNN input modes that may occur with an RNN layer.
If the RNN is configured with RNNInputMode.LINEAR , then for each gate g in the first layer of
the RNN, the input vector X[t] (length E) is left-multiplied by the gate’s corresponding weight ma-
trix W[g] (dimensions HxE) as usual, before being used to compute the gate output as described by
RNNOperation .
If the RNN is configured with RNNInputMode.SKIP , then this initial matrix multiplication is
“skipped” and W[g] is conceptually an identity matrix. In this case, the input vector X[t] must have
length H (the size of the hidden state).
Members:
INPUT : Input Gate
OUTPUT : Output Gate
FORGET : Forget Gate
UPDATE : Update Gate
RESET : Reset Gate
CELL : Cell Gate
HIDDEN : Hidden Gate
class tensorrt.IRNNv2Layer
An RNN layer in an INetworkDefinition , version 2
Variables
• num_layers - int The layer count of the RNN.
• hidden_size - int The hidden size of the RNN.
• max_seq_length - int The maximum sequence length of the RNN.
• data_length - int The embedding length of the RNN.
• seq_lengths - ITensor Individual sequence lengths in the batch with the ITensor pro-
vided. The seq_lengths ITensor should be a {N1, . . . , Np} tensor, where N1..Np are
the index dimensions of the input tensor to the RNN. If seq_lengths is not specified, then
the RNN layer assumes all sequences are size max_seq_length . All sequence lengths in
seq_lengths should be in the range [1, max_seq_length ]. Zero-length sequences are not
supported. This tensor must be of type int32 .
• op - RNNOperation The operation of the RNN layer.
• input_mode - int The input mode of the RNN layer.
• direction - int The direction of the RNN layer.
• hidden_state - ITensor the initial hidden state of the RNN with the provided
hidden_state ITensor . The hidden_state ITensor should have the dimensions {N1,
..., Np, L, H}, where: N1..Np are the index dimensions specified by the input tensor L is the
number of layers in the RNN, equal to num_layers H is the hidden state for each layer, equal
to hidden_size if direction is RNNDirection.UNIDIRECTION , and 2x hidden_size
otherwise.
5.3. Layers
79
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• cell_state - ITensor The initial cell state of the LSTM with the provided cell_state
ITensor . The cell_state ITensor should have the dimensions {N1, . . . , Np, L, H},
where: N1..Np are the index dimensions specified by the input tensor L is the number of layers
in the RNN, equal to num_layers H is the hidden state for each layer, equal to hidden_size
if direction is RNNDirection.UNIDIRECTION, and 2x hidden_size otherwise. It is an
error to set this on an RNN layer that is not configured with RNNOperation.LSTM .
get_bias_for_gate(self: tensorrt.tensorrt.IRNNv2Layer, layer_index: int, gate:
tensorrt.tensorrt.RNNGateType, is_w: bool) → numpy.ndarray
Get the bias parameters for an individual gate in the RNN.
Parameters
• layer_index - The index of the layer that contains this gate.
• gate - The name of the gate within the RNN layer.
• is_w - True if the bias parameters are for the input bias Wb[g] and false if they are for the
recurrent input bias Rb[g].
Returns The bias parameters.
get_weights_for_gate(self: tensorrt.tensorrt.IRNNv2Layer, layer_index: int, gate:
tensorrt.tensorrt.RNNGateType, is_w: bool) → numpy.ndarray
Get the weight parameters for an individual gate in the RNN.
Parameters
• layer_index - The index of the layer that contains this gate.
• gate - The name of the gate within the RNN layer.
• is_w - True if the weight parameters are for the input matrix W[g] and false if they are for
the recurrent input matrix R[g].
Returns The weight parameters.
set_bias_for_gate(self: tensorrt.tensorrt.IRNNv2Layer, layer_index: int, gate:
tensorrt.tensorrt.RNNGateType, is_w: bool, bias: tensorrt.tensorrt.Weights) → None
Set the bias parameters for an individual gate in the RNN.
Parameters
• layer_index - The index of the layer that contains this gate.
• gate - The name of the gate within the RNN layer. The gate name must correspond to one
of the gates used by this layer’s RNNOperation .
• is_w - True if the bias parameters are for the input bias Wb[g] and false if they are for
the recurrent input bias Rb[g]. See RNNOperation for equations showing how these bias
vectors are used in the RNN gate.
• bias - The weight structure holding the bias parameters, which should be an array of size
hidden_size .
set_weights_for_gate(self: tensorrt.tensorrt.IRNNv2Layer, layer_index: int, gate:
tensorrt.tensorrt.RNNGateType, is_w: bool, weights: tensorrt.tensorrt.Weights) →
None
Set the weight parameters for an individual gate in the RNN.
Parameters
• layer_index - The index of the layer that contains this gate.
80
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• gate - The name of the gate within the RNN layer. The gate name must correspond to one
of the gates used by this layer’s RNNOperation .
• is_w - True if the weight parameters are for the input matrix W[g] and false if they are
for the recurrent input matrix R[g]. See RNNOperation for equations showing how these
matrices are used in the RNN gate.
• weights - The weight structure holding the weight parameters, which are stored as a row-
major 2D matrix. For more information, see IRNNv2Layer::setWeights().
5.3.15 IPluginV2Layer
class tensorrt.IPluginV2Layer
A plugin layer in an INetworkDefinition .
Variables plugin - IPluginV2 The plugin for the layer.
5.3.16 IUnaryLayer
tensorrt.UnaryOperation
The unary operations that may be performed by a Unary layer.
Members:
EXP : Exponentiation
LOG : Log (base e)
SQRT : Square root
RECIP : Reciprocal
ABS : Absolute value
NEG : Negation
SIN : Sine
COS : Cosine
TAN : Tangent
SINH : Hyperbolic sine
COSH : Hyperbolic cosine
ASIN : Inverse sine
ACOS : Inverse cosine
ATAN : Inverse tangent
ASINH : Inverse hyperbolic sine
ACOSH : Inverse hyperbolic cosine
ATANH : Inverse hyperbolic tangent
CEIL : Ceiling
FLOOR : Floor
ERF : Gauss error function
5.3. Layers
81
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
NOT : Not
SIGN : Sign. If input > 0, output 1; if input < 0, output -1; if input == 0, output 0.
ROUND : Round to nearest even for floating-point data type.
ISINF : Return true if the input value equals +/- infinity for floating-point data type.
class tensorrt.IUnaryLayer
A unary layer in an INetworkDefinition .
Variables op - UnaryOperation The unary operation for the layer. When running this layer on
DLA, only UnaryOperation.ABS is supported.
5.3.17 IReduceLayer
tensorrt.ReduceOperation
The reduce operations that may be performed by a Reduce layer
Members:
SUM :
PROD :
MAX :
MIN :
AVG :
class tensorrt.IReduceLayer
A reduce layer in an INetworkDefinition .
Variables
• op - ReduceOperation The reduce operation for the layer.
• axes - int The axes over which to reduce.
• keep_dims - bool Specifies whether or not to keep the reduced dimensions for the layer.
5.3.18 IPaddingLayer
class tensorrt.IPaddingLayer
A padding layer in an INetworkDefinition .
Variables
• pre_padding - DimsHW The padding that is applied at the start of the tensor. Negative
padding results in trimming the edge by the specified amount.
• post_padding - DimsHW The padding that is applied at the end of the tensor. Negative
padding results in trimming the edge by the specified amount
• pre_padding_nd - Dims The padding that is applied at the start of the tensor. Negative
padding results in trimming the edge by the specified amount. Only 2 dimensions currently
supported.
• post_padding_nd - Dims The padding that is applied at the end of the tensor. Negative
padding results in trimming the edge by the specified amount. Only 2 dimensions currently
supported.
82
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.19 IParametricReLULayer
class tensorrt.IParametricReLULayer
A parametric ReLU layer in an INetworkDefinition .
This layer applies a parametric ReLU activation to an input tensor (first input), with slopes taken from a slopes
tensor (second input). This can be viewed as a leaky ReLU operation where the negative slope differs from
element to element (and can in fact be learned).
The slopes tensor must be unidirectional broadcastable to the input tensor: the rank of the two tensors must be
the same, and all dimensions of the slopes tensor must either equal the input tensor or be 1. The output tensor
has the same shape as the input tensor.
5.3.20 ISelectLayer
class tensorrt.ISelectLayer
A select layer in an INetworkDefinition .
This layer implements an element-wise ternary conditional operation. Wherever condition is True, elements
are taken from the first input, and wherever condition is False, elements are taken from the second input.
5.3.21 IShuffleLayer
class tensorrt.Permutation(*args, **kwargs)
The elements of the permutation.
The permutation is applied as outputDimensionIndex = permuta-
tion[inputDimensionIndex], so to permute from CHW order to HWC order, the required permutation is [1, 2, 0],
and to permute from HWC to CHW, the required permutation is [2, 0, 1].
It supports iteration and indexing and is implicitly convertible to/from Python iterables (like tuple or list ).
Therefore, you can use those classes in place of Permutation .
Overloaded function.
1.
__init__(self: tensorrt.tensorrt.Permutation) -> None
2.
__init__(self: tensorrt.tensorrt.Permutation, arg0: List[int]) -> None
class tensorrt.IShuffleLayer
A shuffle layer in an INetworkDefinition .
This class shuffles data by applying in sequence: a transpose operation, a reshape operation and a second trans-
pose operation. The dimension types of the output are those of the reshape dimension.
Variables
• first_transpose - Permutation The permutation applied by the first transpose opera-
tion. Default: Identity Permutation
• reshape_dims - Dims The reshaped dimensions. Two special values can be used as di-
mensions. Value 0 copies the corresponding dimension from input. This special value can
be used more than once in the dimensions. If number of reshape dimensions is less than
input, 0s are resolved by aligning the most significant dimensions of input. Value -1 infers
that particular dimension by looking at input and rest of the reshape dimensions. Note that
only a maximum of one dimension is permitted to be specified as -1. The product of the new
dimensions must be equal to the product of the old.
• second_transpose - Permutation The permutation applied by the second transpose op-
eration. Default: Identity Permutation
5.3. Layers
83
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• zero_is_placeholder - bool The meaning of 0 in reshape dimensions. If true, then a 0
in the reshape dimensions denotes copying the corresponding dimension from the first input
tensor. If false, then a 0 in the reshape dimensions denotes a zero-length dimension.
set_input(self: tensorrt.tensorrt.IShuffleLayer, index: int, tensor: tensorrt.tensorrt.ITensor) → None
Sets the input tensor for the given index. The index must be 0 for a static shuffle layer. A static shuffle layer
is converted to a dynamic shuffle layer by calling set_input() with an index 1. A dynamic shuffle layer
cannot be converted back to a static shuffle layer.
For a dynamic shuffle layer, the values 0 and 1 are valid. The indices in the dynamic case are as follows:
Index
Description
0
Data or Shape tensor to be shuffled.
1
The dimensions for the reshape operation, as a 1D int32 shape tensor.
If this function is called with a value 1, then num_inputs changes from 1 to 2.
Parameters
• index - The index of the input tensor.
• tensor - The input tensor.
5.3.22 ISliceLayer
Note: [Deprecated] Use SampleMode instead.
tensorrt.SliceMode
Controls how ISliceLayer and IGridSample handles out of bounds coordinates
Members:
STRICT_BOUNDS : Fail with error when the coordinates are out of bounds.
DEFAULT : [DEPRECATED] Use STRICT_BOUNDS.
WRAP : Coordinates wrap around periodically.
CLAMP : Out of bounds indices are clamped to bounds
FILL : Use fill input value when coordinates are out of bounds.
REFLECT : Coordinates reflect.
class tensorrt.ISliceLayer
A slice layer in an INetworkDefinition .
The slice layer has two variants, static and dynamic. Static slice specifies the start, size, and stride dimensions
at layer creation time via Dims and can use the get/set accessor functions of the ISliceLayer . Dynamic slice
specifies one or more of start, size or stride as ITensor`s, by using :func:`ILayer.set_input to add a
second, third, or fourth input respectively. The corresponding Dims are used if an input is missing or null.
An application can determine if the ISliceLayer has a dynamic output shape based on whether the size input
(third input) is present and non-null.
The slice layer selects for each dimension a start location from within the input tensor, and copies elements to the
output tensor using the specified stride across the input tensor. Start, size, and stride tensors must be 1-D int32
shape tensors if not specified via Dims .
84
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
An example of using slice on a tensor: input = {{0, 2, 4}, {1, 3, 5}} start = {1, 0} size = {1, 2} stride = {1, 2}
output = {{1, 5}}
When the sliceMode is SliceMode.CLAMP or SliceMode.REFLECT , for each input dimension, if its size is 0
then the corresponding output dimension must be 0 too.
A slice layer can produce a shape tensor if the following conditions are met:
• start, size, and stride are build time constants, either as static Dims or as constant input tensors.
• The number of elements in the output tensor does not exceed 2 * Dims.MAX_DIMS .
The input tensor is a shape tensor if the output is a shape tensor.
The following constraints must be satisfied to execute this layer on DLA: * start, size, and stride are build
time constants, either as static Dims or as constant input tensors. * sliceMode is SliceMode.DEFAULT . * Strides
are 1 for all dimensions. * Slicing is not performed on the first dimension * The input tensor has four dimensions
Variables
• start - Dims The start offset.
• shape - Dims The output dimensions.
• stride - Dims The slicing stride.
• mode - SliceMode Controls how ISliceLayer handles out of bounds coordinates.
set_input(self: tensorrt.tensorrt.ISliceLayer, index: int, tensor: tensorrt.tensorrt.ITensor) → None
Sets the input tensor for the given index. The index must be 0 or 4 for a static slice layer. A static slice layer
is converted to a dynamic slice layer by calling set_input() with an index between 1 and 3. A dynamic
slice layer cannot be converted back to a static slice layer.
The indices are as follows:
Index
Description
0
Data or Shape tensor to be sliced.
1
The start tensor to begin slicing, N-dimensional for Data, and 1-D for Shape.
2
The size tensor of the resulting slice, N-dimensional for Data, and 1-D for Shape.
3
The stride of the slicing operation, N-dimensional for Data, and 1-D for Shape.
4
Value for the SliceMode.FILL slice mode. Disallowed for other modes.
If this function is called with a value greater than 0, then num_inputs changes from 1 to index + 1.
Parameters
• index - The index of the input tensor.
• tensor - The input tensor.
5.3.23 IShapeLayer
class tensorrt.IShapeLayer
A shape layer in an INetworkDefinition . Used for getting the shape of a tensor. This class sets the output to
a one-dimensional tensor with the dimensions of the input tensor.
For example, if the input is a four-dimensional tensor (of any type) with dimensions [2,3,5,7], the output tensor
is a one-dimensional int32 tensor of length 4 containing the sequence 2, 3, 5, 7.
5.3. Layers
85
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.24 ITopKLayer
tensorrt.TopKOperation
The operations that may be performed by a TopK layer
Members:
MAX : Maximum of the elements
MIN : Minimum of the elements
class tensorrt.ITopKLayer
A TopK layer in an INetworkDefinition .
Variables
• op - TopKOperation The operation for the layer.
• k-TopKOperation thekvalueforthelayer. Currentlyonlyvaluesupto3840aresupported.
Use the set_input() method with index 1 to pass in dynamic k as a tensor.
• axes - TopKOperation The axes along which to reduce.
set_input(self: tensorrt.tensorrt.ITopKLayer, index: int, tensor: tensorrt.tensorrt.ITensor) → None
Sets the input tensor for the given index. The index must be 0 or 1 for a TopK layer.
The indices are as follows:
Index
Description
0
Input data tensor.
1
A scalar Int32 tensor containing a positive value
corresponding to
of top elements to retrieve. Values larger
than 3840 will result in a runtime error. If
provided, this will override the static k value
in calculations.
Parameters
• index - The index of the input tensor.
• tensor - The input tensor.
5.3.25 IMatrixMultiplyLayer
tensorrt.MatrixOperation
The matrix operations that may be performed by a Matrix layer
Members:
NONE :
TRANSPOSE : Transpose each matrix
VECTOR : Treat operand as collection of vectors
class tensorrt.IMatrixMultiplyLayer
A matrix multiply layer in an INetworkDefinition .
Let A be op(getInput(0)) and B be op(getInput(1)) where op(x) denotes the corresponding MatrixOperation.
86
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
When A and B are matrices or vectors, computes the inner product A * B:
matrix * matrix -> matrix
matrix * vector -> vector
vector * matrix -> vector
vector * vector -> scalar
Inputs of higher rank are treated as collections of matrices or vectors. The output will be a corresponding col-
lection of matrices, vectors, or scalars.
Variables
• op0 - MatrixOperation How to treat the first input.
• op1 - MatrixOperation How to treat the second input.
5.3.26 IRaggedSoftMaxLayer
class tensorrt.IRaggedSoftMaxLayer
A ragged softmax layer in an INetworkDefinition .
This layer takes a ZxS input tensor and an additional Zx1 bounds tensor holding the lengths of the Z sequences.
This layer computes a softmax across each of the Z sequences.
The output tensor is of the same size as the input tensor.
5.3.27 IIdentityLayer
class tensorrt.IIdentityLayer
A layer that represents the identity function.
If tensor precision is explicitly specified, it can be used to transform from one precision to another.
Other than conversions between the same type (float32 -> float32 for example), the only valid conversions
are:
(float32 | float16 | int32 | bool) -> (float32 | float16 | int32 | bool)
(float32 | float16) -> uint8
uint8 -> (float32 | float16)
5.3.28 IConstantLayer
class tensorrt.IConstantLayer
A constant layer in an INetworkDefinition .
Note: This layer does not support boolean and uint8 types.
Variables
• weights - Weights The weights for the layer.
• shape - Dims The shape of the layer.
5.3. Layers
87
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.29 IResizeLayer
tensorrt.ResizeMode
Various modes of interpolation, used in resize and grid_sample layers.
Members:
NEAREST : 1D, 2D, and 3D nearest neighbor interpolation.
LINEAR : Supports linear, bilinear, trilinear interpolation.
CUBIC : Supports bicubic interpolation.
class tensorrt.IResizeLayer
A resize layer in an INetworkDefinition .
Resize layer can be used for resizing a N-D tensor.
Resize layer currently supports the following configurations:
• ResizeMode.NEAREST - resizes innermost m dimensions of N-D, where 0 < m <= min(3, N) and N > 0.
• ResizeMode.LINEAR - resizes innermost m dimensions of N-D, where 0 < m <= min(3, N) and N > 0.
• ResizeMode.CUBIC - resizes innermost 2 dimensions of N-D, N >= 2.
Default resize mode is ResizeMode.NEAREST.
Resize layer provides two ways to resize tensor dimensions:
• Set output dimensions directly. It can be done for static as well as dynamic resize layer. Static resize
layer requires output dimensions to be known at build-time. Dynamic resize layer requires output
dimensions to be set as one of the input tensors.
• Set scales for resize. Each output dimension is calculated as floor(input dimension * scale). Only
static resize layer allows setting scales where the scales are known at build-time.
If executing this layer on DLA, the following combinations of parameters are supported:
• In NEAREST mode:
- (ResizeCoordinateTransformation.ASYMMETRIC,
ResizeSelector.FORMULA,
ResizeRound-
Mode.FLOOR)
- (ResizeCoordinateTransformation.HALF_PIXEL,
ResizeSelector.FORMULA,
ResizeRound-
Mode.HALF_DOWN)
- (ResizeCoordinateTransformation.HALF_PIXEL,
ResizeSelector.FORMULA,
ResizeRound-
Mode.HALF_UP)
• In LINEAR and CUBIC mode:
- (ResizeCoordinateTransformation.HALF_PIXEL, ResizeSelector.FORMULA)
- (ResizeCoordinateTransformation.HALF_PIXEL, ResizeSelector.UPPER)
Variables
• shape - Dims The output dimensions. Must to equal to input dimensions size.
• scales - List[float] List of resize scales. If executing this layer on DLA, there are three
restrictions: 1. len(scales) has to be exactly 4. 2. The first two elements in scales need
to be exactly 1 (for unchanged batch and channel dimensions). 3. The last two elements in
scales, representing the scale values along height and width dimensions, respectively, need
to be integer values in the range of [1, 32] for NEAREST mode and [1, 4] for LINEAR.
Example of DLA-supported scales: [1, 1, 2, 2].
88
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• resize_mode - ResizeMode Resize mode can be Linear, Cubic or Nearest.
• coordinate_transformation - ResizeCoordinateTransformationDoc Supported
resize coordinate transformation modes are ALIGN_CORNERS, ASYMMETRIC and
HALF_PIXEL.
• selector_for_single_pixel - ResizeSelector Supported resize selector modes are
FORMULA and UPPER.
• nearest_rounding - ResizeRoundMode Supported resize Round modes are HALF_UP,
HALF_DOWN, FLOOR and CEIL.
• exclude_outside - int If set to 1, the weight of sampling locations outside the input
tensor will be set to 0, and the weight will be renormalized so that their sum is 1.0.
• cubic_coeff - float coefficient ‘a’ used in cubic interpolation.
set_input(self: tensorrt.tensorrt.IResizeLayer, index: int, tensor: tensorrt.tensorrt.ITensor) → None
Sets the input tensor for the given index.
If index == 1 and num_inputs == 1, and there is no implicit batch dimension, in which case num_inputs
changes to 2. Once such additional input is set, resize layer works in dynamic mode. When index == 1 and
num_inputs == 1, the output dimensions are used from the input tensor, overriding the dimensions supplied
by shape.
Parameters
• index - The index of the input tensor.
• tensor - The input tensor.
5.3.30 ILoop
class tensorrt.ILoop
Helper for creating a recurrent subgraph.
Variables name - The name of the loop. The name is used in error diagnostics.
add_iterator(self: tensorrt.tensorrt.ILoop, tensor: tensorrt.tensorrt.ITensor, axis: int = 0, reverse: bool =
False) → tensorrt.tensorrt.IIteratorLayer
Return layer that subscripts tensor by loop iteration.
For reverse=false, this is equivalent to add_gather(tensor, I, 0) where I is a scalar tensor containing the loop
iteration number. For reverse=true, this is equivalent to add_gather(tensor, M-1-I, 0) where M is the trip
count computed from TripLimits of kind COUNT.
Parameters
• tensor - The tensor to iterate over.
• axis - The axis along which to iterate.
• reverse - Whether to iterate in the reverse direction.
Returns The IIteratorLayer , or None if it could not be created.
add_loop_output(self: tensorrt.tensorrt.ILoop, tensor: tensorrt.tensorrt.ITensor, kind:
tensorrt.tensorrt.LoopOutput, axis: int = 0) → tensorrt.tensorrt.ILoopOutputLayer
Make an output for this loop, based on the given tensor.
If kind is CONCATENATE or REVERSE, a second input specifying the concatenation dimension must be
added via method ILoopOutputLayer.set_input() .
5.3. Layers
89
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Parameters
• kind - The kind of loop output. See LoopOutput
• axis - The axis for concatenation (if using kind of CONCATENATE or REVERSE).
Returns The added ILoopOutputLayer , or None if it could not be created.
add_recurrence(self: tensorrt.tensorrt.ILoop, initial_value: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.IRecurrenceLayer
Create a recurrence layer for this loop with initial_value as its first input.
Parameters initial_value - The initial value of the recurrence layer.
Returns The added IRecurrenceLayer , or None if it could not be created.
add_trip_limit(self: tensorrt.tensorrt.ILoop, tensor: tensorrt.tensorrt.ITensor, kind:
tensorrt.tensorrt.TripLimit) → tensorrt.tensorrt.ITripLimitLayer
Add a trip-count limiter, based on the given tensor.
There may be at most one COUNT and one WHILE limiter for a loop. When both trip limits exist, the loop
exits when the count is reached or condition is falsified. It is an error to not add at least one trip limiter.
For WHILE, the input tensor must be the output of a subgraph that contains only layers that are not
ITripLimitLayer , IIteratorLayer or ILoopOutputLayer . Any IRecurrenceLayer s in the sub-
graph must belong to the same loop as the ITripLimitLayer . A trivial example of this rule is that the
input to the WHILE is the output of an IRecurrenceLayer for the same loop.
Parameters
• tensor - The input tensor. Must be available before the loop starts.
• kind - The kind of trip limit. See TripLimit
Returns The added ITripLimitLayer , or None if it could not be created.
5.3.30.1 ILoopBoundaryLayer
class tensorrt.ILoopBoundaryLayer
Variables loop - ILoop associated with this boundary layer.
5.3.30.1.1 ITripLimitLayer
tensorrt.TripLimit
Describes kinds of trip limits.
Members:
COUNT : Tensor is a scalar of type int32 that contains the trip count.
WHILE : Tensor is a scalar of type bool. Loop terminates when its value is false.
class tensorrt.ITripLimitLayer
Variables kind - The kind of trip limit. See TripLimit
90
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.30.1.2 IRecurrenceLayer
class tensorrt.IRecurrenceLayer
set_input(self: tensorrt.tensorrt.IRecurrenceLayer, index: int, tensor: tensorrt.tensorrt.ITensor) → None
Set the first or second input. If index==1 and the number of inputs is one, the input is appended. The first
input specifies the initial output value, and must come from outside the loop. The second input specifies
the next output value, and must come from inside the loop. The two inputs must have the same dimensions.
Parameters
• index - The index of the input to set.
• tensor - The input tensor.
5.3.30.1.3 IIteratorLayer
class tensorrt.IIteratorLayer
Variables
• axis - The axis to iterate over
• reverse - For reverse=false, the layer is equivalent to add_gather(tensor, I, 0) where I is a
scalar tensor containing the loop iteration number. For reverse=true, the layer is equivalent
to add_gather(tensor, M-1-I, 0) where M is the trip count computed from TripLimits of kind
COUNT. The default is reverse=false.
5.3.30.1.4 ILoopOutputLayer
tensorrt.LoopOutput
Describes kinds of loop outputs.
Members:
LAST_VALUE : Output value is value of tensor for last iteration.
CONCATENATE : Output value is concatenation of values of tensor for each iteration, in forward
order.
REVERSE : Output value is concatenation of values of tensor for each iteration, in reverse order.
class tensorrt.ILoopOutputLayer
An ILoopOutputLayer is the sole way to get output from a loop.
The first input tensor must be defined inside the loop; the output tensor is outside the loop. The second input
tensor, if present, must be defined outside the loop.
If kind is LAST_VALUE, a single input must be provided.
If kind is CONCATENATE or REVERSE, a second input must be provided. The second input must be a scalar “shape
tensor”, defined before the loop commences, that specifies the concatenation length of the output.
The output tensor has j more dimensions than the input tensor, where j == 0 if kind is LAST_VALUE j == 1 if
kind is CONCATENATE or REVERSE.
Variables
• axis - The contenation axis. Ignored if kind is LAST_VALUE. For example, if the input
tensor has dimensions [b,c,d], and kind is CONCATENATE, the output has four dimensions.
Let a be the value of the second input. axis=0 causes the output to have dimensions [a,b,c,d].
5.3. Layers
91
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
axis=1 causes the output to have dimensions [b,a,c,d]. axis=2 causes the output to have
dimensions [b,c,a,d]. axis=3 causes the output to have dimensions [b,c,d,a]. Default is axis
is 0.
• kind - The kind of loop output. See LoopOutput
set_input(self: tensorrt.tensorrt.ILoopOutputLayer, index: int, tensor: tensorrt.tensorrt.ITensor) → None
Like ILayer.set_input(), but additionally works if index==1, num_inputs`==1, in which case
:attr:`num_inputs changes to 2.
5.3.31 IFillLayer
tensorrt.FillOperation
The tensor fill operations that may performed by an Fill layer.
Members:
LINSPACE : Generate evenly spaced numbers over a specified interval
RANDOM_UNIFORM : Generate a tensor with random values drawn from a uniform distribution
RANDOM_NORMAL : Generate a tensor with random values drawn from a normal distribution
class tensorrt.IFillLayer
A fill layer in an INetworkDefinition .
set_input(self: tensorrt.tensorrt.IFillLayer, index: int, tensor: tensorrt.tensorrt.ITensor) → None
replace an input of this layer with a specific tensor.
In-
Description for kLINSPACE
dex
0
Shape tensor, represents the output tensor’s dimensions.
1
Start, a scalar, represents the start value.
2
Delta, a 1D tensor, length equals to shape tensor’s nbDims, represents the delta value for each
dimension.
Index
Description for kRANDOM_UNIFORM
0
Shape tensor, represents the output tensor’s dimensions.
1
Minimum, a scalar, represents the minimum random value.
2
Maximum, a scalar, represents the maximal random value.
Index
Description for kRANDOM_NORMAL
0
Shape tensor, represents the output tensor’s dimensions.
1
Mean, a scalar, represents the mean of the normal distribution.
2
Scale, a scalar, represents the standard deviation of the normal distribution.
Parameters
• index - the index of the input to modify.
• tensor - the input tensor.
92
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.32 IQuantizeLayer
class tensorrt.IQuantizeLayer
A Quantize layer in an INetworkDefinition .
This layer accepts a floating-point data input tensor, and uses the scale and zeroPt inputs to
quantize the data to an 8-bit signed integer according to:
output = clamp(round(input/scale) + zeroP t)
Rounding type is rounding-to-nearest ties-to-even (https://en.wikipedia.org/wiki/Rounding#Round_half_to_
even).
Clamping is in the range [-128, 127].
The first input (index 0) is the tensor to be quantized. The second (index 1) and third (index 2) are the scale and
zero point respectively. Each of scale and zeroPt must be either a scalar, or a 1D tensor.
The zeroPt tensor is optional, and if not set, will be assumed to be zero. Its data type must be tensorrt.int8. zeroPt
must only contain zero-valued coefficients, because only symmetric quantization is supported. The scale value
must be either a scalar for per-tensor quantization, or a 1D tensor for per-axis quantization. The size of the 1-D
scale tensor must match the size of the quantization axis. The size of the scale must match the size of the zeroPt.
The subgraph which terminates with the scale tensor must be a build-time constant. The same restrictions apply
to the zeroPt. The output type, if constrained, must be constrained to tensorrt.int8. The input type, if constrained,
must be constrained to tensorrt.float32 (FP16 input is not supported). The output size is the same as the input
size.
IQuantizeLayer only supports tensorrt.float32 precision and will default to this precision during instantiation.
IQuantizeLayer only supports tensorrt.int8 output.
Variables axis - int The axis along which quantization occurs. The quantization axis is in reference
to the input tensor’s dimensions.
5.3.33 IDequantizeLayer
class tensorrt.IDequantizeLayer
A Dequantize layer in an INetworkDefinition .
This layer accepts a signed 8-bit integer input tensor, and uses the configured scale and zeroPt inputs to dequantize
the input according to: output = (input - zeroP t) * scale
The first input (index 0) is the tensor to be quantized. The second (index 1) and third (index 2) are the scale and
zero point respectively. Each of scale and zeroPt must be either a scalar, or a 1D tensor.
The zeroPt tensor is optional, and if not set, will be assumed to be zero. Its data type must be tensorrt.int8. zeroPt
must only contain zero-valued coefficients, because only symmetric quantization is supported. The scale value
must be either a scalar for per-tensor quantization, or a 1D tensor for per-axis quantization. The size of the 1-D
scale tensor must match the size of the quantization axis. The size of the scale must match the size of the zeroPt.
The subgraph which terminates with the scale tensor must be a build-time constant. The same restrictions apply
to the zeroPt. The output type, if constrained, must be constrained to tensorrt.int8. The input type, if constrained,
must be constrained to tensorrt.float32 (FP16 input is not supported). The output size is the same as the input
size.
IDequantizeLayer only supports tensorrt.int8 precision and will default to this precision during instantiation.
IDequantizeLayer only supports tensorrt.float32 output.
Variables axis - int The axis along which dequantization occurs. The dequantization axis is in
reference to the input tensor’s dimensions.
5.3. Layers
93
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.34 IScatterLayer
class tensorrt.IScatterLayer
A Scatter layer as in INetworkDefinition. :ivar axis: axis to scatter on when using Scatter Element mode
(ignored in ND mode) :ivar mode: ScatterMode The operation mode of the scatter.
5.3.35 IIfConditional
class tensorrt.IIfConditional
Helper for constructing conditionally-executed subgraphs.
An If-conditional conditionally executes (lazy evaluation) part of the network according to the following pseudo-
code:
If condition is true Then:
output = trueSubgraph(trueInputs);
Else:
output = falseSubgraph(falseInputs);
Emit output
Condition is a 0D boolean tensor (representing a scalar). trueSubgraph represents a network subgraph that is
executed when condition is evaluated to True. falseSubgraph represents a network subgraph that is executed
when condition is evaluated to False.
The following constraints apply to If-conditionals: - Both the trueSubgraph and falseSubgraph must be defined.
- The number of output tensors in both subgraphs is the same. - The type and shape of each output tensor from
true/false subgraphs are the same.
add_input(self: tensorrt.tensorrt.IIfConditional, input: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.IIfConditionalInputLayer
Make an input for this if-conditional, based on the given tensor.
Parameters input - An input to the conditional that can be used by either or both of the condi-
tional’s subgraphs.
add_output(self: tensorrt.tensorrt.IIfConditional, true_subgraph_output: tensorrt.tensorrt.ITensor,
false_subgraph_output: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.IIfConditionalOutputLayer
Make an output for this if-conditional, based on the given tensors.
Each output layer of the if-conditional represents a single output of either the true-subgraph or the false-
subgraph of the if-conditional, depending on which subgraph was executed.
Parameters
• true_subgraph_output - The output of the subgraph executed when this conditional’s
condition input evaluates to true.
• false_subgraph_output - The output of the subgraph executed when this conditional’s
condition input evaluates to false.
Returns The IIfConditionalOutputLayer , or None if it could not be created.
set_condition(self: tensorrt.tensorrt.IIfConditional, condition: tensorrt.tensorrt.ITensor) →
tensorrt.tensorrt.IConditionLayer
Set the condition tensor for this If-Conditional construct.
The condition tensor must be a 0D data tensor (scalar) with type bool.
94
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Parameters condition - The condition tensor that will determine which subgraph to execute.
Returns The IConditionLayer , or None if it could not be created.
5.3.36 IConditionLayer
class tensorrt.IConditionLayer
Describes the boolean condition of an if-conditional.
5.3.37 IIfConditionalOutputLayer
class tensorrt.IIfConditionalOutputLayer
Describes kinds of if-conditional outputs.
5.3.38 IIfConditionalInputLayer
class tensorrt.IIfConditionalInputLayer
Describes kinds of if-conditional inputs.
5.3.39 IEinsumLayer
class tensorrt.IEinsumLayer
An Einsum layer in an INetworkDefinition .
This layer implements a summation over the elements of the inputs along dimensions specified by the equation
parameter, based on the Einstein summation convention. The layer can have one or more inputs of rank >= 0.
All the inputs must be of same data type. This layer supports all TensorRT data types except bool. There is one
output tensor of the same type as the input tensors. The shape of output tensor is determined by the equation.
The equation specifies ASCII lower-case letters for each dimension in the inputs in the same order as the di-
mensions, separated by comma for each input. The dimensions labeled with the same subscript must match or
be broadcastable. Repeated subscript labels in one input take the diagonal. Repeating a label across multiple
inputs means that those axes will be multiplied. Omitting a label from the output means values along those axes
will be summed. In implicit mode, the indices which appear once in the expression will be part of the output in
increasing alphabetical order. In explicit mode, the output can be controlled by specifying output subscript labels
by adding an arrow (‘->’) followed by subscripts for the output. For example, “ij,jk->ik” is equivalent to “ij,jk”.
Ellipsis (‘. . . ’) can be used in place of subscripts to broadcast the dimensions. See the TensorRT Developer
Guide for more details on equation syntax.
Many common operations can be expressed using the Einsum equation. For example: Matrix Transpose: ij->ji
Sum: ij-> Matrix-Matrix Multiplication: ik,kj->ij Dot Product: i,i-> Matrix-Vector Multiplication: ik,k->i Batch
Matrix Multiplication: ijk,ikl->ijl Batch Diagonal: . . . ii->. . . i
Note that TensorRT does not support ellipsis or diagonal operations.
Variables equation - str The Einsum equation of the layer. The equation is a comma-separated
list of subscript labels, where each label refers to a dimension of the corresponding tensor.
5.3. Layers
95
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.40 IAssertionLayer
class tensorrt.IAssertionLayer
An assertion layer in an INetworkDefinition .
This layer implements assertions. The input must be a boolean shape tensor. If any element of it is False, a
build-time or run-time error occurs. Asserting equality of input dimensions may help the optimizer.
Variables message - string Message to print if the assertion fails.
5.3.41 IOneHotLayer
class tensorrt.IOneHotLayer
A OneHot layer in a network definition.
The OneHot layer has three input tensors: Indices, Values, and Depth, one output tensor, Output, and an axis
attribute. :ivar indices: is an Int32 tensor that determines which locations in Output to set as on_value. :ivar
values: is a two-element (rank=1) tensor that consists of [off_value, on_value] :ivar depth: is an Int32 shape
tensor of rank 0, which contains the depth (number of classes) of the one-hot encoding. The depth tensor must
be a build-time constant, and its value should be positive. :returns: a tensor with rank = rank(indices)+1, where
the added dimension contains the one-hot encoding. :param axis: specifies to which dimension of the output
one-hot encoding is added.
The data types of Output shall be equal to the Values data type. The output is computed by copying off_values
to all output elements, then setting on_value on the indices specified by the indices tensor.
when axis = 0: output[indices[i, j, k], i, j, k] = on_value for all i, j, k and off_value otherwise.
when axis = -1: output[i, j, k, indices[i, j, k]] = on_value for all i, j, k and off_value otherwise.
5.3.42 INonZeroLayer
class tensorrt.INonZeroLayer
A NonZero layer in an INetworkDefinition .
Computes the indices of the input tensor where the value is non-zero. The returned indices are in row-major
order.
The output shape is always {D, C}, where D is the number of dimensions of the input and C is the number of
non-zero values.
5.3.43 INMSLayer
class tensorrt.INMSLayer
A non-maximum suppression layer in an INetworkDefinition .
Boxes: The input boxes tensor to the layer. This tensor contains the input bounding boxes. It is a linear tensor of
type float32 or float16. It has shape [batchSize, numInputBoundingBoxes, numClasses, 4] if the boxes are
per class, or [batchSize, numInputBoundingBoxes, 4] if the same boxes are to be used for each class.
Scores: The input scores tensor to the layer. This tensor contains the per-box scores. It is a linear tensor of the
same type as the boxes tensor. It has shape [batchSize, numInputBoundingBoxes, numClasses].
MaxOutputBoxesPerClass: The input maxOutputBoxesPerClass tensor to the layer. This tensor contains the
maximum number of output boxes per batch item per class. It is a scalar (0D tensor) of type int32.
96
Chapter 5. Network
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
IoUThreshold is the maximum IoU for selected boxes. It is a scalar (0D tensor) of type float32 in the range
[0.0, 1.0]. It is an optional input with default 0.0. Use set_input() to add this optional tensor.
ScoreThreshold is the value that a box score must exceed in order to be selected. It is a scalar (0D tensor) of type
float32. It is an optional input with default 0.0. Use set_input() to add this optional tensor.
The SelectedIndices output tensor contains the indices of the selected boxes. It is a linear tensor of type
int32. It has shape [NumOutputBoxes, 3].] Each row contains a (batchIndex, classIndex, boxIndex) tuple.
The output boxes are sorted in order of increasing batchIndex and then in order of decreasing score within each
batchIndex. For each batchIndex, the ordering of output boxes with the same score is unspecified. If Max-
OutputBoxesPerClass is a constant input, the maximum number of output boxes is batchSize * numClasses *
min(numInputBoundingBoxes, MaxOutputBoxesPerClass). Otherwise, the maximum number of output boxes
is batchSize * numClasses * numInputBoundingBoxes. The maximum number of output boxes is used to deter-
mine the upper-bound on allocated memory for this output tensor.
The NumOutputBoxes output tensor contains the number of output boxes in selectedIndices. It is a scalar (0D
tensor) of type int32.
The NMS algorithm iterates through a set of bounding boxes and their confidence scores, in decreasing order of
score. Boxes are selected if their score is above a given threshold, and their intersection-over-union (IoU) with
previously selected boxes is less than or equal to a given threshold. This layer implements NMS per batch item
and per class.
For each batch item, the ordering of candidate bounding boxes with the same score is unspecified.
Variables
• bounding_box_format - BoundingBoxFormat The bounding box format used by the
layer. Default is CORNER_PAIRS.
• topk_box_limit - int The maximum number of filtered boxes considered for selection.
Default is 2000 for SM 5.3 and 6.2 devices, and 5000 otherwise. The TopK box limit must
be less than or equal to {2000 for SM 5.3 and 6.2 devices, 5000 otherwise}.
set_input(self: tensorrt.tensorrt.INMSLayer, index: int, tensor: tensorrt.tensorrt.ITensor) → None
Sets the input tensor for the given index. The indices are as follows:
Index
Description
0
The required Boxes tensor.
1
The required Scores tensor.
2
The required MaxOutputBoxesPerClass tensor.
3
The optional IoUThreshold tensor.
4
The optional ScoreThreshold tensor.
If this function is called for an index greater or equal to num_inputs, then afterwards num_inputs returns
index + 1, and any missing intervening inputs are set to null. Note that only optional inputs can be missing.
Parameters
• index - The index of the input tensor.
• tensor - The input tensor.
5.3. Layers
97
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
5.3.44 IReverseSequenceLayer
class tensorrt.IReverseSequenceLayer
A ReverseSequence layer in an INetworkDefinition .
This layer performs batch-wise reversal, which slices the input tensor along the axis batch_axis. For the i-th
slice, the operation reverses the first N elements, specified by the corresponding i-th value in sequence_lens,
along sequence_axis and keeps the remaining elements unchanged. The output tensor will have the same
shape as the input tensor.
Variables
• batch_axis - int The batch axis. Default: 1.
• sequence_axis - int The sequence axis. Default: 0.
5.3.45 INormalizationLayer
class tensorrt.INormalizationLayer
A Normalization layer in an INetworkDefinition .
The normalization layer performs the following operation:
X - input Tensor Y - output Tensor S - scale Tensor B - bias Tensor
Y = (X - Mean(X, axes)) / Sqrt(Variance(X) + epsilon) * S + B
Where Mean(X, axes) is a reduction over a set of axes, and Variance(X) = Mean((X - Mean(X, axes)) ^ 2, axes).
Variables
• epsilon - float The epsilon value used for the normalization calculation. Default: 1e-5F.
• axes - int The reduction axes for the normalization calculation.
• num_groups - int The number of groups to split the channels into for the normalization
calculation. Default: 1.
• compute_precision - DataType The datatype used for the compute precision of this
layer. By default TensorRT will run the normalization computation in DataType.kFLOAT32
even in mixed precision mode regardless of any set builder flags to avoid overflow errors.
ILayer.precision and ILayer.set_output_type can still be set to control input and output types
of this layer. Only DataType.kFLOAT32 and DataType.kHALF are valid for this member.
Default: Datatype.FLOAT.
98
Chapter 5. Network
CHAPTER
SIX
PLUGIN
6.1 IPluginCreator
tensorrt.PluginFieldType
The possible field types for custom layer.
Members:
FLOAT16
FLOAT32
FLOAT64
INT8
INT16
INT32
CHAR
DIMS
UNKNOWN
class tensorrt.PluginField(*args, **kwargs)
Contains plugin attribute field names and associated data. This information can be parsed to decode necessary
plugin metadata
Variables
• name - str Plugin field attribute name.
• data - buffer Plugin field attribute data.
• type - PluginFieldType Plugin field attribute type.
• size - int Number of data entries in the Plugin attribute.
Overloaded function.
1.
__init__(self: tensorrt.tensorrt.PluginField, name: tensorrt.tensorrt.FallbackString = ‘’) -> None
2.
__init__(self: tensorrt.tensorrt.PluginField, name: tensorrt.tensorrt.FallbackString, data: buffer, type: ten-
sorrt.tensorrt.PluginFieldType = <PluginFieldType.UNKNOWN: 8>) -> None
class tensorrt.PluginFieldCollection(*args, **kwargs)
Overloaded function.
1.
__init__(self: tensorrt.tensorrt.PluginFieldCollection) -> None
99
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
2.
__init__(self: tensorrt.tensorrt.PluginFieldCollection, arg0: tensorrt.tensorrt.PluginFieldCollection) ->
None
Copy constructor
3.
__init__(self: tensorrt.tensorrt.PluginFieldCollection, arg0: Iterable) -> None
append(self: tensorrt.tensorrt.PluginFieldCollection, x: nvinfer1::PluginField) → None
Add an item to the end of the list
clear(self: tensorrt.tensorrt.PluginFieldCollection) → None
Clear the contents
extend(*args, **kwargs)
Overloaded function.
1. extend(self: tensorrt.tensorrt.PluginFieldCollection, L: tensorrt.tensorrt.PluginFieldCollection) ->
None
Extend the list by appending all the items in the given list
2. extend(self: tensorrt.tensorrt.PluginFieldCollection, L: Iterable) -> None
Extend the list by appending all the items in the given list
insert(self: tensorrt.tensorrt.PluginFieldCollection, i: int, x: nvinfer1::PluginField) → None
Insert an item at a given position.
pop(*args, **kwargs)
Overloaded function.
1. pop(self: tensorrt.tensorrt.PluginFieldCollection) -> nvinfer1::PluginField
Remove and return the last item
2. pop(self: tensorrt.tensorrt.PluginFieldCollection, i: int) -> nvinfer1::PluginField
Remove and return the item at index i
class tensorrt.IPluginCreator
Plugin creator class for user implemented layers
Variables
• tensorrt_version - int Number of PluginField entries.
• name - str Plugin name.
• plugin_version - str Plugin version.
• field_names - list List of fields that needs to be passed to create_plugin() .
• plugin_namespace - str The namespace of the plugin creator based on the plugin library
it belongs to. This can be set while registering the plugin creator.
create_plugin(self: tensorrt.tensorrt.IPluginCreator, name: str, field_collection:
tensorrt.tensorrt.PluginFieldCollection_) → tensorrt.tensorrt.IPluginV2
Creates a new plugin.
Parameters
• name - The name of the plugin.
• field_collection - The PluginFieldCollection for this plugin.
Returns IPluginV2 or None on failure.
100
Chapter 6. Plugin
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
deserialize_plugin(self: tensorrt.tensorrt.IPluginCreator, name: str, serialized_plugin: buffer) →
tensorrt.tensorrt.IPluginV2
Creates a plugin object from a serialized plugin.
Parameters
• name - Name of the plugin.
• serialized_plugin - A buffer containing a serialized plugin.
Returns A new IPluginV2
6.2 IPluginRegistry
class tensorrt.IPluginRegistry
Registers plugin creators.
Variables
• plugin_creator_list - All the registered plugin creators.
• error_recorder - IErrorRecorder Application-implemented error reporting interface
for TensorRT objects.
• parent_search_enabled - bool variable indicating whether parent search is enabled. De-
fault is True.
deregister_creator(self: tensorrt.tensorrt.IPluginRegistry, creator: tensorrt.tensorrt.IPluginCreator) →
bool
Deregister a previously registered plugin creator.
Since there may be a desire to limit the number of plugins, this function provides a mechanism for removing
plugin creators registered in TensorRT. The plugin creator that is specified by creator is removed from
TensorRT and no longer tracked.
Parameters creator - The IPluginCreator instance.
Returns True if the plugin creator was deregistered, False if it was not found in the registry or
otherwise could not be deregistered.
deregister_library(self: tensorrt.tensorrt.IPluginRegistry, handle: capsule) → None
Deregister plugins associated with a library. Any resources acquired when the library was loaded will be
released.
Arg handle: the plugin library handle to deregister.
get_plugin_creator(self: tensorrt.tensorrt.IPluginRegistry, type: str, version: str, plugin_namespace: str
= '') → tensorrt.tensorrt.IPluginCreator
Return plugin creator based on type and version
Parameters
• type - The type of the plugin.
• version - The version of the plugin.
• plugin_namespace - The namespace of the plugin.
Returns An IPluginCreator .
load_library(self: tensorrt.tensorrt.IPluginRegistry, plugin_path: str) → capsule
Load and register a shared library of plugins.
6.2. IPluginRegistry
101
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Arg plugin_path: the plugin library path.
Returns The loaded plugin library handle. The call will fail and return None if any of the plugins
are already registered.
register_creator(self: tensorrt.tensorrt.IPluginRegistry, creator: tensorrt.tensorrt.IPluginCreator,
plugin_namespace: str = '') → bool
Register a plugin creator.
Parameters
• creator - The IPluginCreator instance.
• plugin_namespace - The namespace of the plugin creator.
Returns False if one with the same type is already registered.
tensorrt.get_plugin_registry() → tensorrt.tensorrt.IPluginRegistry
Return the plugin registry for standard runtime
tensorrt.init_libnvinfer_plugins(logger: capsule, namespace: str) → bool
Initialize and register all the existing TensorRT plugins to the IPluginRegistry with an optional namespace.
The plugin library author should ensure that this function name is unique to the library. This function should be
called once before accessing the Plugin Registry.
Parameters
• logger - Logger to print plugin registration information.
• namespace - Namespace used to register all the plugins in this library.
tensorrt.get_builder_plugin_registry(arg0: nvinfer1::EngineCapability) →
tensorrt.tensorrt.IPluginRegistry
Return the plugin registry used for building engines for the specified runtime
102
Chapter 6. Plugin
CHAPTER
SEVEN
INT8
7.1 IInt8Calibrator
tensorrt.CalibrationAlgoType
Version of calibration algorithm to use.
Members:
LEGACY_CALIBRATION
ENTROPY_CALIBRATION
ENTROPY_CALIBRATION_2
MINMAX_CALIBRATION
class tensorrt.IInt8Calibrator(self: tensorrt.tensorrt.IInt8Calibrator) → None
Application-implemented interface for calibration. Calibration is a step performed by the builder when deciding
suitable scale factors for 8-bit inference. It must also provide a method for retrieving representative images which
the calibration process can use to examine the distribution of activations. It may optionally implement a method
for caching the calibration result for reuse on subsequent runs.
To implement a custom calibrator, ensure that you explicitly instantiate the base class in __init__() :
class MyCalibrator(trt.IInt8Calibrator):
def __init__(self):
trt.IInt8Calibrator.__init__(self)
Variables
• batch_size - int The batch size used for calibration batches.
• algorithm - CalibrationAlgoType The algorithm used by this calibrator.
get_algorithm(self: tensorrt.tensorrt.IInt8Calibrator) → tensorrt.tensorrt.CalibrationAlgoType
Get the algorithm used by this calibrator.
Returns The algorithm used by this calibrator.
get_batch(self: tensorrt.tensorrt.IInt8Calibrator, names: List[str]) → List[int]
Get a batch of input for calibration. The batch size of the input must match the batch size returned by
get_batch_size() .
A possible implementation may look like this:
103
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
def get_batch(names):
try:
# Assume self.batches is a generator that provides batch data.
data = next(self.batches)
# Assume that self.device_input is a device buffer allocated by the␣
˓→
constructor.
cuda.memcpy_htod(self.device_input, data)
return [int(self.device_input)]
except StopIteration:
# When we're out of batches, we return either [] or None.
# This signals to TensorRT that there is no calibration data remaining.
return None
Parameters names - The names of the network inputs for each object in the bindings array.
Returns A list of device memory pointers set to the memory containing each network input
data, or an empty list if there are no more batches for calibration. You can allocate these
device buffers with pycuda, for example, and then cast them to int to retrieve the pointer.
get_batch_size(self: tensorrt.tensorrt.IInt8Calibrator) → int
Get the batch size used for calibration batches.
Returns The batch size.
read_calibration_cache(self: tensorrt.tensorrt.IInt8Calibrator) → buffer
Load a calibration cache.
Calibration is potentially expensive, so it can be useful to generate the calibration data once, then use it
on subsequent builds of the network. The cache includes the regression cutoff and quantile values used
to generate it, and will not be used if these do not match the settings of the current calibrator. However,
the network should also be recalibrated if its structure changes, or the input data set changes, and it is the
responsibility of the application to ensure this.
Reading a cache is just like reading any other file in Python. For example, one possible implementation is:
def read_calibration_cache(self):
# If there is a cache, use it instead of calibrating again. Otherwise,␣
˓→
implicitly return None.
if os.path.exists(self.cache_file):
with open(self.cache_file, "rb") as f:
return f.read()
Returns A cache object or None if there is no data.
write_calibration_cache(self: tensorrt.tensorrt.IInt8Calibrator, cache: buffer) → None
Save a calibration cache.
Writing a cache is just like writing any other buffer in Python. For example, one possible implementation
is:
def write_calibration_cache(self, cache):
with open(self.cache_file, "wb") as f:
f.write(cache)
Parameters cache - The calibration cache to write.
104
Chapter 7. Int8
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
7.2 IInt8LegacyCalibrator
class tensorrt.IInt8LegacyCalibrator(self: tensorrt.tensorrt.IInt8LegacyCalibrator) → None
Extends the IInt8Calibrator class. This calibrator requires user parameterization, and is provided as a fall-
back option if the other calibrators yield poor results.
To implement a custom calibrator, ensure that you explicitly instantiate the base class in __init__() :
class MyCalibrator(trt.IInt8LegacyCalibrator):
def __init__(self):
trt.IInt8LegacyCalibrator.__init__(self)
Variables
• quantile - float The quantile (between 0 and 1) that will be used to select the region
maximum when the quantile method is in use. See the user guide for more details on how
the quantile is used.
• regression_cutoff - float The fraction (between 0 and 1) of the maximum used to
define the regression cutoff when using regression to determine the region maximum. See
the user guide for more details on how the regression cutoff is used
get_algorithm(self: tensorrt.tensorrt.IInt8LegacyCalibrator) → tensorrt.tensorrt.CalibrationAlgoType
Signals that this is the legacy calibrator.
Returns CalibrationAlgoType.LEGACY_CALIBRATION
get_batch(self: tensorrt.tensorrt.IInt8LegacyCalibrator, names: List[str]) → List[int]
Get a batch of input for calibration. The batch size of the input must match the batch size returned by
get_batch_size() .
A possible implementation may look like this:
def get_batch(names):
try:
# Assume self.batches is a generator that provides batch data.
data = next(self.batches)
# Assume that self.device_input is a device buffer allocated by the␣
˓→
constructor.
cuda.memcpy_htod(self.device_input, data)
return [int(self.device_input)]
except StopIteration:
# When we're out of batches, we return either [] or None.
# This signals to TensorRT that there is no calibration data remaining.
return None
Parameters names - The names of the network inputs for each object in the bindings array.
Returns A list of device memory pointers set to the memory containing each network input
data, or an empty list if there are no more batches for calibration. You can allocate these
device buffers with pycuda, for example, and then cast them to int to retrieve the pointer.
get_batch_size(self: tensorrt.tensorrt.IInt8LegacyCalibrator) → int
Get the batch size used for calibration batches.
Returns The batch size.
7.2. IInt8LegacyCalibrator
105
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
read_calibration_cache(self: tensorrt.tensorrt.IInt8LegacyCalibrator) → buffer
Load a calibration cache.
Calibration is potentially expensive, so it can be useful to generate the calibration data once, then use it
on subsequent builds of the network. The cache includes the regression cutoff and quantile values used
to generate it, and will not be used if these do not match the settings of the current calibrator. However,
the network should also be recalibrated if its structure changes, or the input data set changes, and it is the
responsibility of the application to ensure this.
Reading a cache is just like reading any other file in Python. For example, one possible implementation is:
def read_calibration_cache(self):
# If there is a cache, use it instead of calibrating again. Otherwise,␣
˓→
implicitly return None.
if os.path.exists(self.cache_file):
with open(self.cache_file, "rb") as f:
return f.read()
Returns A cache object or None if there is no data.
write_calibration_cache(self: tensorrt.tensorrt.IInt8LegacyCalibrator, cache: buffer) → None
Save a calibration cache.
Writing a cache is just like writing any other buffer in Python. For example, one possible implementation
is:
def write_calibration_cache(self, cache):
with open(self.cache_file, "wb") as f:
f.write(cache)
Parameters cache - The calibration cache to write.
7.3 IInt8EntropyCalibrator
class tensorrt.IInt8EntropyCalibrator(self: tensorrt.tensorrt.IInt8EntropyCalibrator) → None
Extends the IInt8Calibrator class.
To implement a custom calibrator, ensure that you explicitly instantiate the base class in __init__() :
class MyCalibrator(trt.IInt8EntropyCalibrator):
def __init__(self):
trt.IInt8EntropyCalibrator.__init__(self)
This is the Legacy Entropy calibrator. It is less complicated than the legacy calibrator and produces better results.
get_algorithm(self: tensorrt.tensorrt.IInt8EntropyCalibrator) → tensorrt.tensorrt.CalibrationAlgoType
Signals that this is the entropy calibrator.
Returns CalibrationAlgoType.ENTROPY_CALIBRATION
get_batch(self: tensorrt.tensorrt.IInt8EntropyCalibrator, names: List[str]) → List[int]
Get a batch of input for calibration. The batch size of the input must match the batch size returned by
get_batch_size() .
A possible implementation may look like this:
106
Chapter 7. Int8
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
def get_batch(names):
try:
# Assume self.batches is a generator that provides batch data.
data = next(self.batches)
# Assume that self.device_input is a device buffer allocated by the␣
˓→
constructor.
cuda.memcpy_htod(self.device_input, data)
return [int(self.device_input)]
except StopIteration:
# When we're out of batches, we return either [] or None.
# This signals to TensorRT that there is no calibration data remaining.
return None
Parameters names - The names of the network inputs for each object in the bindings array.
Returns A list of device memory pointers set to the memory containing each network input
data, or an empty list if there are no more batches for calibration. You can allocate these
device buffers with pycuda, for example, and then cast them to int to retrieve the pointer.
get_batch_size(self: tensorrt.tensorrt.IInt8EntropyCalibrator) → int
Get the batch size used for calibration batches.
Returns The batch size.
read_calibration_cache(self: tensorrt.tensorrt.IInt8EntropyCalibrator) → buffer
Load a calibration cache.
Calibration is potentially expensive, so it can be useful to generate the calibration data once, then use it
on subsequent builds of the network. The cache includes the regression cutoff and quantile values used
to generate it, and will not be used if these do not match the settings of the current calibrator. However,
the network should also be recalibrated if its structure changes, or the input data set changes, and it is the
responsibility of the application to ensure this.
Reading a cache is just like reading any other file in Python. For example, one possible implementation is:
def read_calibration_cache(self):
# If there is a cache, use it instead of calibrating again. Otherwise,␣
˓→
implicitly return None.
if os.path.exists(self.cache_file):
with open(self.cache_file, "rb") as f:
return f.read()
Returns A cache object or None if there is no data.
write_calibration_cache(self: tensorrt.tensorrt.IInt8EntropyCalibrator, cache: buffer) → None
Save a calibration cache.
Writing a cache is just like writing any other buffer in Python. For example, one possible implementation
is:
def write_calibration_cache(self, cache):
with open(self.cache_file, "wb") as f:
f.write(cache)
Parameters cache - The calibration cache to write.
7.3. IInt8EntropyCalibrator
107
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
7.4 IInt8EntropyCalibrator2
class tensorrt.IInt8EntropyCalibrator2(self: tensorrt.tensorrt.IInt8EntropyCalibrator2) → None
Extends the IInt8Calibrator class.
To implement a custom calibrator, ensure that you explicitly instantiate the base class in __init__() :
class MyCalibrator(trt.IInt8EntropyCalibrator2):
def __init__(self):
trt.IInt8EntropyCalibrator2.__init__(self)
This is the preferred calibrator. This is the required calibrator for DLA, as it supports per activation tensor scaling.
get_algorithm(self: tensorrt.tensorrt.IInt8EntropyCalibrator2) → tensorrt.tensorrt.CalibrationAlgoType
Signals that this is the entropy calibrator 2.
Returns CalibrationAlgoType.ENTROPY_CALIBRATION_2
get_batch(self: tensorrt.tensorrt.IInt8EntropyCalibrator2, names: List[str]) → List[int]
Get a batch of input for calibration. The batch size of the input must match the batch size returned by
get_batch_size() .
A possible implementation may look like this:
def get_batch(names):
try:
# Assume self.batches is a generator that provides batch data.
data = next(self.batches)
# Assume that self.device_input is a device buffer allocated by the␣
˓→
constructor.
cuda.memcpy_htod(self.device_input, data)
return [int(self.device_input)]
except StopIteration:
# When we're out of batches, we return either [] or None.
# This signals to TensorRT that there is no calibration data remaining.
return None
Parameters names - The names of the network inputs for each object in the bindings array.
Returns A list of device memory pointers set to the memory containing each network input
data, or an empty list if there are no more batches for calibration. You can allocate these
device buffers with pycuda, for example, and then cast them to int to retrieve the pointer.
get_batch_size(self: tensorrt.tensorrt.IInt8EntropyCalibrator2) → int
Get the batch size used for calibration batches.
Returns The batch size.
read_calibration_cache(self: tensorrt.tensorrt.IInt8EntropyCalibrator2) → buffer
Load a calibration cache.
Calibration is potentially expensive, so it can be useful to generate the calibration data once, then use it
on subsequent builds of the network. The cache includes the regression cutoff and quantile values used
to generate it, and will not be used if these do not match the settings of the current calibrator. However,
the network should also be recalibrated if its structure changes, or the input data set changes, and it is the
responsibility of the application to ensure this.
108
Chapter 7. Int8
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Reading a cache is just like reading any other file in Python. For example, one possible implementation is:
def read_calibration_cache(self):
# If there is a cache, use it instead of calibrating again. Otherwise,␣
˓→implicitly return None.
if os.path.exists(self.cache_file):
with open(self.cache_file, "rb") as f:
return f.read()
Returns A cache object or None if there is no data.
write_calibration_cache(self: tensorrt.tensorrt.IInt8EntropyCalibrator2, cache: buffer) → None
Save a calibration cache.
Writing a cache is just like writing any other buffer in Python. For example, one possible implementation
is:
def write_calibration_cache(self, cache):
with open(self.cache_file, "wb") as f:
f.write(cache)
Parameters cache - The calibration cache to write.
7.5 IInt8MinMaxCalibrator
class tensorrt.IInt8MinMaxCalibrator(self: tensorrt.tensorrt.IInt8MinMaxCalibrator) → None
Extends the IInt8Calibrator class.
To implement a custom calibrator, ensure that you explicitly instantiate the base class in __init__() :
class MyCalibrator(trt.IInt8MinMaxCalibrator):
def __init__(self):
trt.IInt8MinMaxCalibrator.__init__(self)
This is the preferred calibrator for NLP tasks for all backends. It supports per activation tensor scaling.
get_algorithm(self: tensorrt.tensorrt.IInt8MinMaxCalibrator) → tensorrt.tensorrt.CalibrationAlgoType
Signals that this is the minmax calibrator.
Returns CalibrationAlgoType.MINMAX_CALIBRATION
get_batch(self: tensorrt.tensorrt.IInt8MinMaxCalibrator, names: List[str]) → List[int]
Get a batch of input for calibration. The batch size of the input must match the batch size returned by
get_batch_size() .
A possible implementation may look like this:
def get_batch(names):
try:
# Assume self.batches is a generator that provides batch data.
data = next(self.batches)
# Assume that self.device_input is a device buffer allocated by the␣
˓→constructor.
cuda.memcpy_htod(self.device_input, data)
(continues on next page)
7.5. IInt8MinMaxCalibrator
109
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
(continued from previous page)
return [int(self.device_input)]
except StopIteration:
# When we're out of batches, we return either [] or None.
# This signals to TensorRT that there is no calibration data remaining.
return None
Parameters names - The names of the network inputs for each object in the bindings array.
Returns A list of device memory pointers set to the memory containing each network input
data, or an empty list if there are no more batches for calibration. You can allocate these
device buffers with pycuda, for example, and then cast them to int to retrieve the pointer.
get_batch_size(self: tensorrt.tensorrt.IInt8MinMaxCalibrator) → int
Get the batch size used for calibration batches.
Returns The batch size.
read_calibration_cache(self: tensorrt.tensorrt.IInt8MinMaxCalibrator) → buffer
Load a calibration cache.
Calibration is potentially expensive, so it can be useful to generate the calibration data once, then use it
on subsequent builds of the network. The cache includes the regression cutoff and quantile values used
to generate it, and will not be used if these do not match the settings of the current calibrator. However,
the network should also be recalibrated if its structure changes, or the input data set changes, and it is the
responsibility of the application to ensure this.
Reading a cache is just like reading any other file in Python. For example, one possible implementation is:
def read_calibration_cache(self):
# If there is a cache, use it instead of calibrating again. Otherwise,␣
˓→
implicitly return None.
if os.path.exists(self.cache_file):
with open(self.cache_file, "rb") as f:
return f.read()
Returns A cache object or None if there is no data.
write_calibration_cache(self: tensorrt.tensorrt.IInt8MinMaxCalibrator, cache: buffer) → None
Save a calibration cache.
Writing a cache is just like writing any other buffer in Python. For example, one possible implementation
is:
def write_calibration_cache(self, cache):
with open(self.cache_file, "wb") as f:
f.write(cache)
Parameters cache - The calibration cache to write.
110
Chapter 7. Int8
CHAPTER
EIGHT
ALGORITHM SELECTOR
class tensorrt.IAlgorithmIOInfo
This class carries information about input or output of the algorithm. IAlgorithmIOInfo for all the input and
output along with IAlgorithmVariant denotes the variation of algorithm and can be used to select or reproduce
an algorithm using IAlgorithmSelector.select_algorithms().
Variables
• tensor_format - TensorFormat [DEPRECATED] TensorFormat of the input/output of
algorithm. This is deprecated since the strides, data type, and vectorization information is
sufficient to uniquely identify tensor formats.
• dtype - DataType DataType of the input/output of algorithm.
• strides - Dims strides of the input/output tensor of algorithm.
• vectorized_dim - int the index of the vectorized dimension or -1 for non-vectorized
formats.
• components_per_element - int the number of components per element. This is always
1 for non-vectorized formats.
__init__(*args, **kwargs)
class tensorrt.IAlgorithmVariant
provides a unique 128-bit identifier, which along with the input and output information denotes the variation of
algorithm and can be used to select or reproduce an algorithm, using IAlgorithmSelector.select_algorithms() see
IAlgorithmIOInfo, IAlgorithm, IAlgorithmSelector.select_algorithms() note A single implementation can have
multiple tactics.
Variables
• implementation - int implementation of the algorithm.
• tactic - int tactic of the algorithm.
__init__(*args, **kwargs)
class tensorrt.IAlgorithmContext
Describes the context and requirements, that could be fulfilled by one or more instances of IAlgorithm. see
IAlgorithm
Variables
• name - str name of the algorithm node.
• num_inputs - int number of inputs of the algorithm.
• num_outputs - int number of outputs of the algorithm.
__init__(*args, **kwargs)
111
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
get_shape(self: tensorrt.tensorrt.IAlgorithmContext, index: int) → List[tensorrt.tensorrt.Dims]
Get the minimum / optimum / maximum dimensions for a dynamic input tensor.
Parameters index - Index of the input or output of the algorithm. Incremental numbers assigned
to indices of inputs and the outputs.
Returns A List[Dims] of length 3, containing the minimum, optimum, and maximum shapes, in
that order. If the shapes have not been set yet, an empty list is returned.`
class tensorrt.IAlgorithm
Application-implemented interface for selecting and reporting the tactic selection of a layer. Tactic
Selection is a step performed by the builder for deciding best algorithms for a layer.
Variables
• algorithm_variant - IAlgorithmVariant& the algorithm variant.
• timing_msec - float The time in milliseconds to execute the algorithm.
• workspace_size - int The size of the GPU temporary memory in bytes which the algo-
rithm uses at execution time.
__init__(*args, **kwargs)
get_algorithm_io_info(self: tensorrt.tensorrt.IAlgorithm, index: int) →
tensorrt.tensorrt.IAlgorithmIOInfo
A single call for both inputs and outputs. Incremental numbers assigned to indices of inputs and the outputs.
Parameters index - Index of the input or output of the algorithm. Incremental numbers assigned
to indices of inputs and the outputs.
Returns A IAlgorithmIOInfo&
class tensorrt.IAlgorithmSelector(self: tensorrt.tensorrt.IAlgorithmSelector) → None
Interface implemented by application for selecting and reporting algorithms of a layer provided by the builder.
note A layer in context of algorithm selection may be different from ILayer in INetworkDefiniton. For example,
an algorithm might be implementing a conglomeration of multiple ILayers in INetworkDefinition.
To implement a custom algorithm selector, ensure that you explicitly instantiate the base class in __init__() :
class MyAlgoSelector(trt.IAlgorithmSelector):
def __init__(self):
trt.IAlgorithmSelector.__init__(self)
__init__(self: tensorrt.tensorrt.IAlgorithmSelector) → None
report_algorithms(self: tensorrt.tensorrt.IAlgorithmSelector, contexts:
List[tensorrt.tensorrt.IAlgorithmContext], choices: List[tensorrt.tensorrt.IAlgorithm])
→ None
Called by TensorRT to report choices it made.
Note: For a given optimization profile, this call comes after all calls to select_algorithms. choices[i] is the
choice that TensorRT made for algoContexts[i], for i in [0, num_algorithms-1]
For example, a possible implementation may look like this:
def report_algorithms(self, contexts, choices):
# Prints the time of the chosen algorithm by TRT from the
# selection list passed in by select_algorithms
(continues on next page)
112
Chapter 8. Algorithm Selector
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
(continued from previous page)
for choice in choices:
print(choice.timing_msec)
Parameters
• contexts - The list of all algorithm contexts.
• choices - The list of algorithm choices made by TensorRT corresponding to each context.
select_algorithms(self: tensorrt.tensorrt.IAlgorithmSelector, context:
tensorrt.tensorrt.IAlgorithmContext, choices: List[tensorrt.tensorrt.IAlgorithm]) →
List[int]
Select Algorithms for a layer from the given list of algorithm choices.
Note: TRT uses its default algorithm selection to choose from the list returned by the user. If the returned
list is empty, TRT’s default algorithm selection is used unless strict type constraints are set. The list of
choices is valid only for this specific algorithm context.
For example, the simplest implementation looks like this:
def select_algorithms(self, context, choices):
assert len(choices) > 0
return list(range(len(choices)))
Parameters
• context - The context for which the algorithm choices are valid.
• choices - The list of algorithm choices to select for implementation of this layer.
Returns A List[int] indicating the indices from the choices vector that TensorRT should
choose from.
113
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
114
Chapter 8. Algorithm Selector
CHAPTER
NINE
UFF PARSER
tensorrt.UffInputOrder
The different possible supported input orders.
Members:
NCHW
NHWC
NC
class tensorrt.UffParser(self: tensorrt.tensorrt.UffParser) → None
This class is used for parsing models described using the UFF format.
Variables
• uff_required_version_major - int Version Major of the UFF.
• uff_required_version_minor - int Version Minor of the UFF.
• uff_required_version_patch - int Version Patch of the UFF.
• plugin_namespace - str The namespace used to lookup and create plugins in the network.
• error_recorder - IErrorRecorder Application-implemented error reporting interface
for TensorRT objects.
__del__(self: tensorrt.tensorrt.UffParser) → None
__exit__(exc_type, exc_value, traceback)
Context managers are deprecated and have no effect. Objects are automatically freed when the reference
count reaches 0.
__init__(self: tensorrt.tensorrt.UffParser) → None
parse(self: tensorrt.tensorrt.UffParser, file: str, network: tensorrt.tensorrt.INetworkDefinition, weights_type:
tensorrt.tensorrt.DataType = <DataType.FLOAT: 0>) → bool
Parse a UFF file.
Parameters
• file - File name of the UFF file.
• network - Network in which the UffParser will fill the layers.
• weights_type - The type on which the weights will be transformed in.
Returns True if the UFF file is parsed without error.
115
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
parse_buffer(self: tensorrt.tensorrt.UffParser, buffer: buffer, network: tensorrt.tensorrt.INetworkDefinition,
weights_type: tensorrt.tensorrt.DataType = <DataType.FLOAT: 0>) → bool
Parse a UFF buffer - useful if the file is already live in memory.
Parameters
• buffer - The UFF buffer.
• network - Network in which the UFFParser will fill the layers.
• weights_type - The type on which the weights will be transformed in.
Returns True if the UFF buffer is parsed without error.
register_input(self: tensorrt.tensorrt.UffParser, name: str, shape: tensorrt.tensorrt.Dims, order:
tensorrt.tensorrt.UffInputOrder = <UffInputOrder.NCHW: 0>) → bool
Register an input name of a UFF network with the associated Dimensions.
Parameters
• name - Input name.
• shape - Input shape.
• order - Input order on which the framework input was originally.
Returns True if the name registers without error.
register_output(self: tensorrt.tensorrt.UffParser, name: str) → bool
Register an output name of a UFF network.
Parameters output_name - Output name.
Returns True if the name registers without error.
9.1 Fields
tensorrt.FieldType
The possible field types for the custom layer.
Members:
FLOAT
INT32
CHAR
DIMS
DATATYPE
UNKNOWN
class tensorrt.FieldMap(self: tensorrt.tensorrt.FieldMap, name: str, data: capsule, type:
tensorrt.tensorrt.FieldType, length: int = 1) → None
This is a class containing an array of field params used as a layer parameter for plugin layers. The node fields are
passed by the parser to the API through the plugin constructor. The implementation of the plugin should parse
the contents of the FieldMap as part of the plugin constructor.
Variables
• name - str field param
• data - capsule field param
116
Chapter 9. UFF Parser
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• type - FieldType field param
• length - int field param
class tensorrt.FieldCollection
This class contains an array of FieldMap s.
Variables
• num_fields - int The number of FieldMap s.
• fields - capsule The array of FieldMap s.
9.1. Fields
117
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
118
Chapter 9. UFF Parser
CHAPTER
TEN
CAFFE PARSER
class tensorrt.IBlobNameToTensor
This class is used to store and query ITensor s after they have been extracted from a Caffe model using the
CaffeParser .
find(self: tensorrt.tensorrt.IBlobNameToTensor, name: str) → tensorrt.tensorrt.ITensor
Given a blob name, this function returns an ITensor object.
Parameters name - Caffe blob name for which the user wants the corresponding ITensor .
Returns A ITensor object corresponding to the queried name. If no such ITensor exists, then
an empty object is returned.
class tensorrt.CaffeParser(self: tensorrt.tensorrt.CaffeParser) → None
This class is used for parsing Caffe models. It allows users to export models trained using Caffe to TRT.
Variables
• plugin_factory_v2 - ICaffePluginFactoryV2 The ICaffePluginFactory used to create
the user defined plugins.
• plugin_namespace - str The namespace used to lookup and create plugins in the network.
• protobuf_buffer_size - int The buffer size for the parsing and storage of the learned
model.
• error_recorder - IErrorRecorder Application-implemented error reporting interface
for TensorRT objects.
__del__(self: tensorrt.tensorrt.CaffeParser) → None
__exit__(exc_type, exc_value, traceback)
Context managers are deprecated and have no effect. Objects are automatically freed when the reference
count reaches 0.
__init__(self: tensorrt.tensorrt.CaffeParser) → None
parse(self: tensorrt.tensorrt.CaffeParser, deploy: str, model: str, network:
tensorrt.tensorrt.INetworkDefinition, dtype: tensorrt.tensorrt.DataType) →
tensorrt.tensorrt.IBlobNameToTensor
Parse a prototxt file and a binaryproto Caffe model to extract network definition and weights associated
with the network, respectively.
Parameters
• deploy - The plain text, prototxt file used to define the network definition.
• model - The binaryproto Caffe model that contains the weights associated with the net-
work.
119
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• network - Network in which the CaffeParser will fill the layers.
• dtype - The type to which the weights will be transformed.
Returns An IBlobNameToTensor object that contains the extracted data.
parse_binary_proto(self: tensorrt.tensorrt.CaffeParser, filename: str) → numpy.ndarray
Parse and extract data stored in binaryproto file. The binaryproto file contains data stored in a binary blob.
parse_binary_proto() converts it to an numpy.ndarray object.
Parameters filename - Path to file containing binary proto.
Returns numpy.ndarray An array that contains the extracted data.
parse_buffer(self: tensorrt.tensorrt.CaffeParser, deploy_buffer: buffer, model_buffer: buffer, network:
tensorrt.tensorrt.INetworkDefinition, dtype: tensorrt.tensorrt.DataType) →
tensorrt.tensorrt.IBlobNameToTensor
Parse a prototxt file and a binaryproto Caffe model to extract network definition and weights associated
with the network, respectively.
Parameters
• deploy_buffer - The memory buffer containing the plain text deploy prototxt used to
define the network definition.
• model_buffer - The binaryproto Caffe memory buffer that contains the weights associ-
ated with the network.
• network - Network in which the CaffeParser will fill the layers.
• dtype - The type to which the weights will be transformed.
Returns An IBlobNameToTensor object that contains the extracted data.
tensorrt.shutdown_protobuf_library() → None
Shuts down protocol buffers library.
10.1 Plugins
class tensorrt.ICaffePluginFactoryV2
Plugin factory used to configure plugins.
create_plugin(self: tensorrt.tensorrt.ICaffePluginFactoryV2, layer_name: str, weights:
std::vector<nvinfer1::Weights, std::allocator<nvinfer1::Weights> >) →
tensorrt.tensorrt.IPluginV2
Creates a plugin.
arg layer_name Name of layer associated with the plugin.
arg weights Weights used for the layer.
Returns The newly created IPluginV2 .
is_plugin_v2(self: tensorrt.tensorrt.ICaffePluginFactoryV2, layer_name: str) → bool
A user implemented function that determines if a layer configuration is provided by an IPluginV2 .
Parameters layer_name - Name of the layer which the user wishes to validate.
Returns True if the layer configuration is provided by an IPluginV2 .
120
Chapter 10. Caffe Parser
CHAPTER
ELEVEN
ONNX PARSER
class tensorrt.OnnxParser(self: tensorrt.tensorrt.OnnxParser, network: tensorrt.tensorrt.INetworkDefinition,
logger: tensorrt.tensorrt.ILogger) → None
This class is used for parsing ONNX models into a TensorRT network definition
Variables num_errors - int The number of errors that occurred during prior calls to parse()
Parameters
• network - The network definition to which the parser will write.
• logger - The logger to use.
__del__(self: tensorrt.tensorrt.OnnxParser) → None
__exit__(exc_type, exc_value, traceback)
Context managers are deprecated and have no effect. Objects are automatically freed when the reference
count reaches 0.
__init__(self: tensorrt.tensorrt.OnnxParser, network: tensorrt.tensorrt.INetworkDefinition, logger:
tensorrt.tensorrt.ILogger) → None
Parameters
• network - The network definition to which the parser will write.
• logger - The logger to use.
clear_errors(self: tensorrt.tensorrt.OnnxParser) → None
Clear errors from prior calls to parse()
clear_flag(self: tensorrt.tensorrt.OnnxParser, flag: nvonnxparser::OnnxParserFlag) → None
Clears the parser flag from the enabled flags.
Parameters flag - The flag to clear.
get_error(self: tensorrt.tensorrt.OnnxParser, index: int) → nvonnxparser::IParserError
Get an error that occurred during prior calls to parse()
Parameters index - Index of the error
get_flag(self: tensorrt.tensorrt.OnnxParser, flag: nvonnxparser::OnnxParserFlag) → bool
Check if a build mode flag is set.
Parameters flag - The flag to check.
Returns A bool indicating whether the flag is set.
121
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
get_used_vc_plugin_libraries(self: tensorrt.tensorrt.OnnxParser) → List[str]
Query the plugin libraries needed to implement operations used by the parser in a version-compatible
engine.
This provides a list of plugin libraries on the filesystem needed to implement operations in the parsed
network. If you are building a version-compatible engine using this network, provide this list to IBuilder-
Config.set_plugins_to_serialize() to serialize these plugins along with the version-compatible engine, or, if
you want to ship these plugin libraries externally to the engine, ensure that IPluginRegistry.load_library()
is used to load these libraries in the appropriate runtime before deserializing the corresponding engine.
Returns List[str] List of plugin libraries found by the parser.
Raises RuntimeError if an internal error occurred when trying to fetch the list of plugin li-
braries.
parse(self: tensorrt.tensorrt.OnnxParser, model: buffer, path: str = None) → bool
Parse a serialized ONNX model into the TensorRT network.
Parameters
• model - The serialized ONNX model.
• path - The path to the model file. Only required if the model has externally stored weights.
Returns true if the model was parsed successfully
parse_from_file(self: tensorrt.tensorrt.OnnxParser, model: str) → bool
Parse an ONNX model from file into a TensorRT network.
Parameters model - The path to an ONNX model.
Returns true if the model was parsed successfully
parse_with_weight_descriptors(self: tensorrt.tensorrt.OnnxParser, model: buffer) → bool
Parse a serialized ONNX model into the TensorRT network with consideration of user provided weights.
Parameters model - The serialized ONNX model.
Returns true if the model was parsed successfully
set_flag(self: tensorrt.tensorrt.OnnxParser, flag: nvonnxparser::OnnxParserFlag) → None
Add the input parser flag to the already enabled flags.
Parameters flag - The flag to set.
supports_model(self: tensorrt.tensorrt.OnnxParser, model: buffer, path: str = None) → Tuple[bool,
List[Tuple[List[int], bool]]]
Check whether TensorRT supports a particular ONNX model.
Parameters
• model - The serialized ONNX model.
• path - The path to the model file. Only required if the model has externally stored weights.
Returns Tuple[bool, List[Tuple[NodeIndices, bool]]] The first element of the tuple indicates
whether the model is supported. The second indicates subgraphs (by node index) in the model
and whether they are supported.
supports_operator(self: tensorrt.tensorrt.OnnxParser, op_name: str) → bool
Returns whether the specified operator may be supported by the parser. Note that a result of true does not
guarantee that the operator will be supported in all cases (i.e., this function may return false-positives).
Parameters op_name - The name of the ONNX operator to check for support
122
Chapter 11. Onnx Parser
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
tensorrt.ErrorCode
The type of parser error
Members:
SUCCESS
INTERNAL_ERROR
MEM_ALLOC_FAILED
MODEL_DESERIALIZE_FAILED
INVALID_VALUE
INVALID_GRAPH
INVALID_NODE
UNSUPPORTED_GRAPH
UNSUPPORTED_NODE
class tensorrt.ParserError
code(self: tensorrt.tensorrt.ParserError) → tensorrt.tensorrt.ErrorCode
Returns The error code
desc(self: tensorrt.tensorrt.ParserError) → str
Returns Description of the error
file(self: tensorrt.tensorrt.ParserError) → str
Returns Source file in which the error occurred
func(self: tensorrt.tensorrt.ParserError) → str
Returns Source function in which the error occurred
line(self: tensorrt.tensorrt.ParserError) → int
Returns Source line at which the error occurred
node(self: tensorrt.tensorrt.ParserError) → int
Returns Index of the Onnx model node in which the error occurred
123
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
124
Chapter 11. Onnx Parser
CHAPTER
TWELVE
UFF CONVERTER
The uff package contains a set of utilites to convert trained models from various frameworks to a common format.
12.1 Conversion Tools
12.1.1 Tensorflow Modelstream to UFF
uff.from_tensorflow(graphdef, output_nodes=[], preprocessor=None, **kwargs)
Converts a TensorFlow GraphDef to a UFF model.
Parameters
• graphdef (tensorflow.GraphDef ) - The TensorFlow graph to convert.
• output_nodes (list(str)) - The names of the outputs of the graph. If not provided,
graphsurgeon is used to automatically deduce output nodes.
• output_filename (str) - The UFF file to write.
• preprocessor (str) - The path to a preprocessing script that will be executed before the
converter. This script should define a preprocess function which accepts a graphsurgeon
DynamicGraph and modifies it in place.
• write_preprocessed (bool) - If set to True, the converter will write out the prepro-
cessed graph as well as a TensorBoard visualization. Must be used in conjunction with
output_filename.
• text (bool) - If set to True, the converter will also write out a human readable UFF file.
Must be used in conjunction with output_filename.
• quiet (bool) - If set to True, suppresses informational messages. Errors may still be
printed.
• debug_mode (bool) - If set to True, the converter prints verbose debug messages.
• return_graph_info (bool) - If set to True, this function returns the graph input and output
nodes in addition to the serialized UFF graph.
Returns
serialized UFF MetaGraph (str)
OR, if return_graph_info is set to True,
serialized UFF MetaGraph (str), graph inputs (list(tensorflow.NodeDef)), graph outputs
(list(tensorflow.NodeDef))
125
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
12.1.2 Tensorflow Frozen Protobuf Model to UFF
uff.from_tensorflow_frozen_model(frozen_file, output_nodes=[], preprocessor=None, **kwargs)
Converts a TensorFlow frozen graph to a UFF model.
Parameters
• frozen_file (str) - The path to the frozen TensorFlow graph to convert.
• output_nodes (list(str)) - The names of the outputs of the graph. If not provided,
graphsurgeon is used to automatically deduce output nodes.
• output_filename (str) - The UFF file to write.
• preprocessor (str) - The path to a preprocessing script that will be executed before the
converter. This script should define a preprocess function which accepts a graphsurgeon
DynamicGraph and modifies it in place.
• write_preprocessed (bool) - If set to True, the converter will write out the prepro-
cessed graph as well as a TensorBoard visualization. Must be used in conjunction with
output_filename.
• text (bool) - If set to True, the converter will also write out a human readable UFF file.
Must be used in conjunction with output_filename.
• quiet (bool) - If set to True, suppresses informational messages. Errors may still be
printed.
• debug_mode (bool) - If set to True, the converter prints verbose debug messages.
• return_graph_info (bool) - If set to True, this function returns the graph input and output
nodes in addition to the serialized UFF graph.
Returns
serialized UFF MetaGraph (str)
OR, if return_graph_info is set to True,
serialized UFF MetaGraph (str), graph inputs (list(tensorflow.NodeDef)), graph outputs
(list(tensorflow.NodeDef))
126
Chapter 12. UFF Converter
CHAPTER
THIRTEEN
UFF OPERATORS
All shapes include batch dimension, unless otherwise specified.
13.1 Input
An input to the network. Expects a CHW shape.
13.1.1 Supported Datatypes
float32, float16, int32, int8
13.2 Identity
Identity layer.
13.2.1 Inputs
Input0 [Tensor or Constant] Input0 to the identity.
13.2.2 Supported Datatypes
float32. float16, int32, int8
13.3 Const
A constant in the network. Should not include batch dimension.
127
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.3.1 Supported Datatypes
float32, float16, int32, int8
13.4 Conv
A convolution operation.
13.4.1 Inputs
Input0 [Tensor] The input to the convolution. Must be 4 dimensional. Automatically transposed to
NCHW.
Kernel [Constant] The kernel weights for the convolution. Must be 4 dimensional. Automatically trans-
posed to NCHW.
13.4.2 Attributes
dilation [List[int]] The HW dilations.
strides [List[int]] The HW strides.
padding [List[int]] The HW padding. Asymmetric padding is unsupported.
13.4.3 Supported Datatypes
float32, float16, int8
13.5 ConvTranspose
A transposed convolution, also known as deconvolution.
13.5.1 Inputs
Input0 [Tensor] The input to the transposed convolution. Must be 4 dimensional. Automatically trans-
posed to NCHW.
Kernel [Constant] The kernel weights for the transposed convolution. Must be 4 dimensional. Automat-
ically transposed to NCHW.
Shape [Constant] The HW dimensions of the output.
128
Chapter 13. UFF Operators
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.5.2 Attributes
strides [List[int]] The HW strides.
padding [List[int]] The HW padding. Asymmetric padding is unsupported.
13.5.3 Supported Datatypes
float32, float16, int8
13.6 Pool
A pooling layer.
13.6.1 Inputs
Input0 [Tensor] The input to the pooling layer. Must be 4 dimensional.
13.6.2 Attributes
func [Enum[max, avg]] The type of pooling to apply.
kernel [List[int]] The HW shape of the kernel.
strides [List[int]] The HW strides.
padding [List[int]] The HW padding.
13.6.3 Supported Datatypes
float32, float16, int8
13.7 FullyConnected
A fully connected layer.
13.7.1 Inputs
Input0 [Tensor] The input to the fully connected layer. Must be at least 4 dimensional. Automatically
transposed to -NC-.
Weights [Constant] The weights for the fully connected layer. Must be 3 dimensional. Automatically
transposed to CHW, where C is the number of output channels.
13.6. Pool
129
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.7.2 Supported Datatypes
float32, float16, int8
13.8 LRN
An LRN layer.
13.8.1 Inputs
Input0 [Tensor] The input to the LRN. Must be at least 4 dimensional.
13.8.2 Attributes
window_size [int] The window size.
alpha [double] The LRN alpha value.
beta [double] The LRN beta value.
k [double] The LRN k value.
13.8.3 Supported Datatypes
float32, float16, int8
13.9 Binary
A binary layer.
13.9.1 Inputs
Input0 [Tensor or Constant] The first input to the binary layer.
Input1 [Tensor or Constant] The second input to the binary layer.
If either input is a constant, then at least one of the inputs must be 4 dimensional.
13.9.2 Attributes
func [Enum[min, max, mul, sub, div, add, pow]] The type of operation to perform.
130
Chapter 13. UFF Operators
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.9.3 Supported Datatypes
float32, float16, int32, int8
13.10 Unary
A unary layer.
13.10.1 Inputs
Input0 [Tensor or Constant] The input to the unary layer.
The output of a unary layer with a Constant input is treated as a Constant, and therefore will not work with layers
expecting a Tensor input.
13.10.2 Attributes
func [Enum[neg, exp, log, abs, sqrt, rsqrt, square, sin, cos, tan, sinh, cosh, asin, acos, atan, asinh, acosh, atanh, ceil, floor]]
The type of operation to perform.
13.10.3 Supported Datatypes
float32, float16, int32, int8
13.11 Reshape
A reshape layer. NOTE: this layer destroys order information. Therefore, subsequent layers will cease to automatically
transpose their inputs to the correct format.
13.11.1 Inputs
Input0 [Tensor or Constant] The input to the reshape layer.
Shape [Constant] The desired shape. If the shape has fewer than 3 non-batch dimensions, 1s are inserted
in the least significant dimensions. For example, if the shape specified is [1, 300, 5], it will be treated
as [1, 300, 5, 1] instead. - -1 specifies that the dimension should be automatically deduced - this can
only be used at most once in any given shape. - 0 specifies that the dimension should be copied from
the input.
The output of a reshape layer with a Constant input is treated as a Constant, and therefore will not work with layers
expecting a Tensor input.
13.10. Unary
131
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.11.2 Supported Datatypes
float32, float16, int32, int8
13.12 ExpandDims
An unsqueeze layer. NOTE: this layer destroys order information. Therefore, subsequent layers will cease to automat-
ically transpose their inputs to the correct format.
13.12.1 Inputs
Input0 [Tensor] The input to unsqueeze.
If the resulting output has fewer than 3 non-batch dimensions, it is unsqueezed further with additional 1s inserted in
the least significant dimensions. For example, with an input of shape [1, 300], and axis of 1, the shape of the output
will be [1, 300, 1, 1] rather than [1, 300, 1].
13.12.2 Attributes
axis [int] The axis on which to unsqueeze, excluding batch dimension. For example, unsqueezing an
NCHW tensor with an axis of 0 will result in a N1CHW tensor.
13.12.3 Supported Datatypes
float32, float16, int32, int8
13.13 ArgMax
An argmax layer.
13.13.1 Inputs
Input0 [Tensor] The input to the argmax layer.
13.13.2 Attributes
axis [int] The axis on which to perform the argmax, with 0 corresponding to the batch dimension. The
specified dimension is removed. For example, performing argmax on an input of shape [1, 300, 150],
with an axis of 1 would result in an output of shape [1, 150]. NOTE: argmax on the batch dimension
is not supported.
132
Chapter 13. UFF Operators
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.13.3 Supported Datatypes
float32, float16, int8
13.14 ArgMin
An argmin layer.
13.14.1 Inputs
Input0 [Tensor] The input to the argmin layer.
13.14.2 Attributes
axis [int] The axis on which to perform the argmin, with 0 corresponding to the batch dimension. The
specified dimension is removed. For example, performing argmin on an input of shape [1, 300, 150],
with an axis of 1 would result in an output of shape [1, 150]. NOTE: argmin on the batch dimension
is not supported.
13.14.3 Supported Datatypes
float32, float16, int8
13.15 Transpose
A transpose layer. A no-op in the network, this layer only modifies the UFF parser’s internal order information. There-
fore, when followed by any layer that destroys order information, the transpose will not be performed.
13.15.1 Inputs
Input0 [Tensor] The input to the transpose layer.
13.15.2 Attributes
permutation [int] The permutation to perform. Must be 4 dimensional.
13.15.3 Supported Datatypes
float32, float16, int32, int8
13.14. ArgMin
133
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.16 Reduce
A reduce layer. NOTE: this layer destroys order information. Therefore, subsequent layers will cease to automatically
transpose their inputs to the correct format.
13.16.1 Inputs
Input0 [Tensor or Constant] The input to the reduce layer.
The output of a reduce layer with a Constant input is treated as a Constant, and therefore will not work with layers
expecting a Tensor input.
13.16.2 Attributes
func [Enum[sum, prod, max, min, mean]] The reduction operation to perform.
axes [List[int]] The axes on which to reduce, with 0 corresponding to the batch dimension. Reduction on
the batch dimension is unsupported.
keepdims [bool] Whether to keep the dimensions which were reduced. NOTE: The UFF parser ignored
this value, and always keeps dimensions.
13.16.3 Supported Datatypes
float32, float16, int32, int8
13.17 Concat
A concatenation layer.
13.17.1 Inputs
Inputs (variadic) [List[Tensor]] The tensors to concatenate. All inputs are transposed to the same format
as the first input. Inputs must be at least 4 dimensional.
13.17.2 Attributes
axis [int] The axis on which to perform the concatenation, with 0 corresponding to the batch dimension.
Concatenating on the batch dimension is unsupported.
134
Chapter 13. UFF Operators
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.17.3 Supported Datatypes
float32, float16, int32, int8
13.18 MarkOutput
The output of the network.
13.18.1 Inputs
Inputs (variadic) [List[Tensor]] The inputs to this layer. Automatically transposed to the same order as
the outputs of the original TensorFlow network.
13.18.2 Supported Datatypes
float32, float16, int32, int8
13.19 Activation
An activation layer.
13.19.1 Inputs
Input0 [Tensor] The input to the activation.
13.19.2 Attributes
func [Enum[relu, relu6, sigmoid, tanh, elu, selu, softsign, softplus]] The operation to perform.
13.19.3 Supported Datatypes
float32, float16, int8
13.20 Softmax
A softmax layer.
13.18. MarkOutput
135
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.20.1 Inputs
Input0 [Tensor] The input to the softmax.
13.20.2 Attributes
axis [int] The axis on which to perform the reduction. NOTE: This value is ignored by the UFF parser.
13.20.3 Supported Datatypes
float32, float16, int8
13.21 BatchNorm
A batchnorm layer.
13.21.1 Inputs
Input0 [Tensor] The input to the batchnorm. Must be 4 dimensional.
Gamma [Constant] The gamma values.
Beta [Constant] The beta values.
Mean [Constant] The mean values.
Variance [Constant] The variance values.
13.21.2 Attributes
epsilon [double] The epsilon value.
13.21.3 Supported Datatypes
float32, float16, int8
13.22 Shape
A shape layer. Returns the shape of its input. NOTE: the output of this layer is a Constant, and therefore will not work
with layers expecting a Tensor input.
136
Chapter 13. UFF Operators
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.22.1 Inputs
Input0 [Tensor] The input to the shape layer.
13.22.2 Supported Datatypes
float32, float16, int32, int8
13.23 StridedSlice
A strided slice layer.
13.23.1 Inputs
Input0 [Tensor or Constant] The input to the strided slice.
Begin [Constant] The indices at which to begin slicing.
End [Constant] The indices at which to end slicing.
Strides [Constant] Strides to use when slicing.
13.23.2 Attributes
begin_mask [int] See TensorFlow stridedSlice documentation.
end_mask [int] See TensorFlow stridedSlice documentation.
shrink_axis_mask [int] See TensorFlow stridedSlice documentation. This value is ignored unless the
input is a constant.
13.23.3 Supported Datatypes
float32, float16, int32, int8
13.24 Stack
A stack layer. NOTE: the output of this layer is a Constant, and therefore will not work with layers expecting a Tensor
input.
13.23. StridedSlice
137
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.24.1 Inputs
Inputs (variadic) [List[Constant]] The inputs to the stack layer.
13.24.2 Attributes
axis [int] The axis on which to stack. NOTE: this value is ignored by the UFF parser.
13.24.3 Supported Datatypes
float32, float16, int32, int8
13.25 Squeeze
Not implemented
13.26 Flatten
A flatten layer. A no-op in the UFF parser.
13.26.1 Inputs
Input0 [Tensor] The tensor to flatten.
13.26.2 Supported Datatypes
float32, float16, int32, int8
13.27 Pad
A padding layer.
13.27.1 Inputs
Input0 [Tensor or Constant] The input to pad. The input is automatically transposed if padding is ap-
plied to non-HW dimensions.
Padding [Constant] The padding to apply. Padding is supported on 2 dimensions at most.
The output of a padding layer with a Constant input is treated as a Constant, and therefore will not work with layers
expecting a Tensor input.
138
Chapter 13. UFF Operators
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
13.27.2 Supported Datatypes
float32, float16, int32, int8
13.28 Gather
A gather layer.
13.28.1 Inputs
Input0 [Tensor or Constant] The input to the gather layer. If the input is constant, it is assumed to be of
shape NC11.
Indices [Tensor or Constant] The indices to gather along. These are assumed to be on the first non-batch
dimension.
13.28.2 Supported Datatypes
float32, float16, int32, int8
13.29 GatherV2
A gatherV2 layer.
13.29.1 Inputs
Input0 [Tensor or Constant] The input to the gather layer. If the input is constant, it is assumed to be of
shape NC11.
Indices [Tensor or Constant] The indices to gather along.
13.29.2 Attributes
axis [int] The axis along which to gather, excluding batch dimension.
13.29.3 Supported Datatypes
float32, float16, int32, int8
13.28. Gather
139
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
140
Chapter 13. UFF Operators
CHAPTER
FOURTEEN
GRAPH SURGEON
graphsurgeon allows you to transform TensorFlow graphs. Its capabilities are broadly divided into two categories:
search and manipulation. Search functions allow you to find nodes in a TensorFlow graph. Manipulation functions
allow you to modify, add, or remove nodes.
14.1 Node Creation
Allow you to create free standing TensorFlow nodes, which can be used as stand-ins for plugins.
graphsurgeon.create_node(name, op=None, trt_plugin=False, **kwargs)
Creates a free-standing TensorFlow NodeDef with the specified properties.
Parameters
• name (str) - The name of the node.
• op (str) - The node’s operation.
Keyword Arguments
• dtype (tensorflow.DType) - TensorFlow dtype.
• shape (tuple(int)) - Iterable container (usually a tuple) describing the shape of a tensor.
• inputs (list(tensorflow.NodeDef ) or str) - Iterable container (usually a tuple) of
input nodes or input node names. Supports mixed-type lists.
•
**kwargs (AttrName=Value) - Any additional fields that should be present in the node.
Currently supports int, float, bool, list(int), list(float), str and NumPy arrays. NumPy arrays
will be inserted into the “value” attribute of the node - this can be useful for creating constant
nodes equivalent to those created by tensorflow.constant.
Returns tensorflow.NodeDef
graphsurgeon.create_plugin_node(name, op=None, **kwargs)
Creates a free-standing TensorFlow NodeDef with the specified properties. This is similar to create_node,
Parameters
• name (str) - The name of the node.
• op (str) - The node’s operation.
• dtype (tensorflow.DType) - TensorFlow dtype.
• shape (tuple(int)) - Iterable container (usually a tuple) describing the shape of a tensor.
• inputs (list(tensorflow.NodeDef ) or str) - Iterable container (usually a tuple) of
input nodes or input node names. Supports mixed-type lists.
141
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
•
**kwargs (AttrName=Value) - Any additional fields that should be present in the node.
Currently supports int, float, bool, list(int), list(float) and str.
Returns tensorflow.NodeDef
14.2 Static Graph
class graphsurgeon.StaticGraph(graphdef=None)
Acts as a thin wrapper for a read-only TensorFlow GraphDef. Supports indexing based on node name/index as
well as iteration over nodes using Python’s for node in static_graph syntax.
Parameters graphdef
(tensorflow.GraphDef/tensorflow.Graph OR graphsurgeon.
StaticGraph/graphsurgeon.DynamicGraph OR str) - A TensorFlow GraphDef/Graph or
a StaticGraph from which to construct this graph, or a string containing a path to a frozen model.
node_outputs
A mapping of node names to their respective output nodes.
Type dict(str, list(tensorflow.NodeDef))
node_map
A mapping of node names to their corresponding nodes.
Type dict(str, tensorflow.NodeDef)
graph_outputs
A list of likely outputs of the graph.
Type list(tensorflow.NodeDef)
graph_inputs
A list of likely inputs of the graph.
Type list(tensorflow.NodeDef)
as_graph_def()
Returns this StaticGraph’s internal TensorFlow GraphDef.
Parameters None -
Returns tensorflow.GraphDef
find_node_chains_by_op(chain)
Finds groups of nodes in this graph that match the specified sequence of ops. Returns a list of matching
chains of nodes, with ordering preserved.
Parameters chain (list(str)) - The sequence of ops to look for. Should be ordered with the
input of the chain as the first element, and the output as the last.
Returns list(list(tensorflow.NodeDef))
find_node_inputs(node)
Finds input nodes of a given node.
Parameters node (tensorflow.NodeDef ) - The node in which to perform the search.
Returns list(tensorflow.NodeDef)
find_node_inputs_by_name(node, name)
Finds input nodes of a given node based on their names.
Parameters
142
Chapter 14. Graph Surgeon
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
• node (tensorflow.NodeDef ) - The node in which to perform the search.
• name (str OR list(str)) - The name to look for. Also accepts iterable containers
(preferably a list) to search for multiple names in a single pass. Supports regular expres-
sions.
Returns list(tensorflow.NodeDef)
find_node_inputs_by_op(node, op)
Finds input nodes of a given node based on their ops.
Parameters
• node (tensorflow.NodeDef ) - The node in which to perform the search.
• op (str OR list(str)) - The op to look for. Also accepts iterable containers (preferably
a list) to search for multiple op in a single pass.
Returns list(tensorflow.NodeDef)
find_nodes_by_name(name)
Finds nodes in this graph based on their names.
Parameters name (str OR list(str)) - The name to look for. Also accepts iterable contain-
ers (preferably a list) to search for multiple names in a single pass of the graph. Supports
regular expressions.
Returns list(tensorflow.NodeDef)
find_nodes_by_op(op)
Finds nodes in this graph based on their ops.
Parameters op (str OR set(str)) - The op to look for. Also accepts iterable containers
(preferably hashsets) to search for multiple ops in a single pass of the graph.
Returns list(tensorflow.NodeDef)
find_nodes_by_path(path)
Finds nodes in this graph based on their full paths. This will only match exact paths.
Parameters path (str OR list(str)) - The path to look for. Also accepts iterable containers
(preferably a list) to search for multiple paths in a single pass of the graph. Supports regular
expressions.
Returns list(tensorflow.NodeDef)
read(filename)
Reads a frozen protobuf file into this StaticGraph.
Parameters filename (str) - Name of the protobuf file.
Returns None
write(filename)
Writes the StaticGraph’s internal TensorFlow GraphDef into a frozen protobuf file.
Parameters filename (str) - Name of the protobuf file to write.
Returns None
write_tensorboard(logdir)
Writes the StaticGraph’s internal TensorFlow GraphDef into the specified directory, which can then be
visualized in TensorBoard.
Parameters logdir (str) - Name of the directory to write.
14.2. Static Graph
143
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Returns None
Raises
• Warning - Passing a GraphDef to the SummaryWriter is deprecated. Pass a Graph object
instead, such as sess.graph.
• This is a known warning, but currently there is no alternative,
since TensorFlow will not be able to convert invalid GraphDefs back
to Graphs. -
14.3 Dynamic Graph (Inherits from StaticGraph)
class graphsurgeon.DynamicGraph(graphdef=None)
A sub-class of StaticGraph that can search and modify a TensorFlow GraphDef.
Parameters graphdef
(tensorflow.GraphDef/tensorflow.Graph OR graphsurgeon.
StaticGraph/graphsurgeon.DynamicGraph OR str) - A TensorFlow GraphDef/Graph or
a StaticGraph/DynamicGraph from which to construct this graph, or a string containing the path
to a frozen model.
append(node)
Appends a node to this graph.
Parameters node (tensorflow.NodeDef ) - TensorFlow NodeDef to add to the graph.
Returns None
collapse_namespaces(namespace_map, exclude_nodes=[], unique_inputs=True)
Collapses nodes in namespaces to single nodes specified by the user, except where those nodes are marked
for exclusion.
Parameters
• namespace_map (dict(str, tensorflow.NodeDef )) - A dictionary specifying
namespaces and their corresponding plugin nodes. These plugin nodes are typically used to
specify attributes of the custom plugin, while inputs and outputs are automatically deduced.
Multiple namespaces can be collapsed into a single plugin node, and nested namespaces
are collapsed into plugin nodes outside their parent namespaces.
• exclude_nodes (list(tensorflow.NodeDef )) - Iterable container (usually a list) of
nodes which should NOT be collapsed. These nodes will be present in the final graph as
either inputs or outputs of the plugin nodes.
• unique_inputs (bool) - Whether inputs to the collapsed node should be unique. If this
is false, plugin nodes may have duplicate inputs.
Returns None
extend(node_list)
Extends this graph’s nodes based on the provided list.
Parameters node_list (list(tensorflow.NodeDef )) - List of TensorFlow NodeDefs to
add to the graph.
Returns None
forward_inputs(nodes)
Removes nodes from this graph. Recursively forwards inputs, such that paths in the graph are preserved.
144
Chapter 14. Graph Surgeon
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
Warning: Nodes with control inputs are not removed, so as not to break the structure of the graph. If you
need to forward these, remove their control inputs first.
Parameters nodes (list(tensorflow.NodeDef ))) - Iterable container (usually a list) of
nodes which should be removed and whose inputs forwarded.
Returns None
remove(nodes, remove_exclusive_dependencies=False)
Removes nodes from this graph. Does not forward inputs, so paths in the graph could be broken.
Parameters
• nodes (list(tensorflow.NodeDef ))) - Iterable container (usually a list) of nodes
which should be removed.
• remove_exclusive_dependencies (bool) - Whether to also remove dependencies ex-
clusive to the nodes about to be removed. When set to True, all exclusive dependencies
will be removed recursively, and the number of hanging nodes in the graph will remain
constant. Defaults to False.
Returns None
14.3. Dynamic Graph (Inherits from StaticGraph)
145
NVIDIA TensorRT Standard Python API Documentation, Release 8.6.11
146
Chapter 14. Graph Surgeon
|
||
|
|
|