|
|
9.51 nvinfer1::safe::ICudaEngine Class Reference
257
Returns
The vector component size. -1 will be returned if (1) name is not the name of an input or output tensor, or (2)
name is nullptr, or (3) name exceeds the string length limit, or (4) the tensor of given name is not vectorized.
See also
safe::ICudaEngine::getTensorVectorizedDim()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.20 getTensorDataType()
virtual DataType nvinfer1::safe::ICudaEngine::getTensorDataType (
AsciiChar const ∗ tensorName ) const [pure virtual], [noexcept]
Determine the required data type for a buffer from its tensor name.
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
Returns
The type of the data in the buffer. DataType::kFLOAT will be returned if (1) name is not the name of an input or
output tensor, or (2) name is nullptr, or (3) name exceeds the string length limit.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
258
Class Documentation
9.51.3.21 getTensorFormat()
virtual TensorFormat nvinfer1::safe::ICudaEngine::getTensorFormat (
AsciiChar const ∗ tensorName ) const [pure virtual], [noexcept]
Return the tensor format.
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
Returns
The tensor format. TensorFormat::kLINEAR will be returned if (1) name is not the name of an input or output
tensor, or (2) name is nullptr, or (3) name exceeds the string length limit.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.22 getTensorIOMode()
virtual TensorIOMode nvinfer1::safe::ICudaEngine::getTensorIOMode (
AsciiChar const ∗ tensorName ) const [pure virtual], [noexcept]
Determine whether a tensor is an input or output tensor.
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
9.51 nvinfer1::safe::ICudaEngine Class Reference
259
Returns
kINPUT if tensorName is an input, kOUTPUT if tensorName is an output, or kNONE if neither.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.23 getTensorShape()
virtual Dims nvinfer1::safe::ICudaEngine::getTensorShape (
AsciiChar const ∗ tensorName ) const [pure virtual], [noexcept]
Get extent of an input or output tensor.
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
Returns
Extent of the tensor. Dims{-1, {}} will be returned if (1) name is not the name of an input or output tensor, or
(2) name is nullptr, or (3) name exceeds the string length limit.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.24 getTensorVectorizedDim()
virtual std::int32 t nvinfer1::safe::ICudaEngine::getTensorVectorizedDim (
AsciiChar const ∗ tensorName ) const [pure virtual], [noexcept]
Return the dimension index along which buffer is vectorized.
Specifically -1 is returned if scalars per vector is 1.
260
Class Documentation
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
Returns
The dimension index along which the buffer is vectorized. -1 will be returned if (1) name is not the name of an
input or output tensor, or (2) name is nullptr, or (3) name exceeds the string length limit, or (4) the tensor of given
name is not vectorized.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.25 operator=() [1/2]
ICudaEngine & nvinfer1::safe::ICudaEngine::operator= (
ICudaEngine &&
) & [delete]
9.51.3.26 operator=() [2/2]
ICudaEngine & nvinfer1::safe::ICudaEngine::operator= (
ICudaEngine const &
) & [delete]
9.51.3.27 setErrorRecorder()
virtual void nvinfer1::safe::ICudaEngine::setErrorRecorder (
IErrorRecorder ∗const recorder ) [pure virtual], [noexcept]
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function
will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder
with the interface, resulting in a call to decRefCount if a recorder has been registered.
9.52 nvinfer1::IDeconvolutionLayer Class Reference
261
Parameters
recorder
The error recorder to register with this interface.
See also
getErrorRecorder()
Usage considerations
• Allowed context for the API call
- Thread-safe: No
The documentation for this class was generated from the following file:
• NvInferSafeRuntime.h
9.52
nvinfer1::IDeconvolutionLayer Class Reference
A deconvolution layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IDeconvolutionLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IDeconvolutionLayer
Public Member Functions
• TRT DEPRECATED void setKernelSize (DimsHW kernelSize) noexcept
Set the HW kernel size of the convolution.
• TRT DEPRECATED DimsHW getKernelSize () const noexcept
Get the HW kernel size of the deconvolution.
• void setNbOutputMaps (int32 t nbOutputMaps) noexcept
Set the number of output feature maps for the deconvolution.
• int32 t getNbOutputMaps () const noexcept
Get the number of output feature maps for the deconvolution.
262
Class Documentation
•
TRT DEPRECATED void setStride (DimsHW stride) noexcept
Set the stride of the deconvolution.
•
TRT DEPRECATED DimsHW getStride () const noexcept
Get the stride of the deconvolution.
•
TRT DEPRECATED void setPadding (DimsHW padding) noexcept
Set the padding of the deconvolution.
•
TRT DEPRECATED DimsHW getPadding () const noexcept
Get the padding of the deconvolution.
•
void setNbGroups (int32 t nbGroups) noexcept
Set the number of groups for a deconvolution.
•
int32 t getNbGroups () const noexcept
Get the number of groups for a deconvolution.
•
void setKernelWeights (Weights weights) noexcept
Set the kernel weights for the deconvolution.
•
Weights getKernelWeights () const noexcept
Get the kernel weights for the deconvolution.
•
void setBiasWeights (Weights weights) noexcept
Set the bias weights for the deconvolution.
•
Weights getBiasWeights () const noexcept
Get the bias weights for the deconvolution.
•
void setPrePadding (Dims padding) noexcept
Set the multi-dimension pre-padding of the deconvolution.
•
Dims getPrePadding () const noexcept
Get the pre-padding.
•
void setPostPadding (Dims padding) noexcept
Set the multi-dimension post-padding of the deconvolution.
•
Dims getPostPadding () const noexcept
Get the padding.
•
void setPaddingMode (PaddingMode paddingMode) noexcept
Set the padding mode.
•
PaddingMode getPaddingMode () const noexcept
Get the padding mode.
•
void setKernelSizeNd (Dims kernelSize) noexcept
Set the multi-dimension kernel size of the deconvolution.
•
Dims getKernelSizeNd () const noexcept
Get the multi-dimension kernel size of the deconvolution.
•
void setStrideNd (Dims stride) noexcept
Set the multi-dimension stride of the deconvolution.
•
Dims getStrideNd () const noexcept
Get the multi-dimension stride of the deconvolution.
•
void setPaddingNd (Dims padding) noexcept
Set the multi-dimension padding of the deconvolution.
•
Dims getPaddingNd () const noexcept
Get the multi-dimension padding of the deconvolution.
•
void setDilationNd (Dims dilation) noexcept
Set the multi-dimension dilation of the deconvolution.
•
Dims getDilationNd () const noexcept
Get the multi-dimension dilation of the deconvolution.
•
void setInput (int32 t index, ITensor &tensor) noexcept
Append or replace an input of this layer with a specific tensor.
9.52 nvinfer1::IDeconvolutionLayer Class Reference
263
Protected Member Functions
• virtual ∼IDeconvolutionLayer () noexcept=default
Protected Attributes
• apiv::VDeconvolutionLayer ∗ mImpl
9.52.1
Detailed Description
A deconvolution layer in a network definition.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.52.2
Constructor & Destructor Documentation
9.52.2.1
∼IDeconvolutionLayer()
virtual nvinfer1::IDeconvolutionLayer::∼IDeconvolutionLayer ( ) [protected], [virtual], [default],
[noexcept]
9.52.3
Member Function Documentation
9.52.3.1
getBiasWeights()
Weights nvinfer1::IDeconvolutionLayer::getBiasWeights ( ) const [inline], [noexcept]
Get the bias weights for the deconvolution.
See also
getBiasWeights()
264
Class Documentation
9.52.3.2
getDilationNd()
Dims nvinfer1::IDeconvolutionLayer::getDilationNd ( ) const [inline], [noexcept]
Get the multi-dimension dilation of the deconvolution.
See also
setDilationNd()
9.52.3.3
getKernelSize()
TRT DEPRECATED DimsHW nvinfer1::IDeconvolutionLayer::getKernelSize ( ) const [inline], [noexcept]
Get the HW kernel size of the deconvolution.
See also
setKernelSize()
Deprecated Superseded by getKernelSizeNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.52.3.4
getKernelSizeNd()
Dims nvinfer1::IDeconvolutionLayer::getKernelSizeNd ( ) const [inline], [noexcept]
Get the multi-dimension kernel size of the deconvolution.
See also
setKernelSizeNd()
9.52.3.5
getKernelWeights()
Weights nvinfer1::IDeconvolutionLayer::getKernelWeights ( ) const [inline], [noexcept]
Get the kernel weights for the deconvolution.
See also
setNbGroups()
9.52 nvinfer1::IDeconvolutionLayer Class Reference
265
9.52.3.6
getNbGroups()
int32 t nvinfer1::IDeconvolutionLayer::getNbGroups ( ) const [inline], [noexcept]
Get the number of groups for a deconvolution.
See also
setNbGroups()
9.52.3.7
getNbOutputMaps()
int32 t nvinfer1::IDeconvolutionLayer::getNbOutputMaps ( ) const [inline], [noexcept]
Get the number of output feature maps for the deconvolution.
See also
setNbOutputMaps()
9.52.3.8
getPadding()
TRT DEPRECATED DimsHW nvinfer1::IDeconvolutionLayer::getPadding ( ) const [inline], [noexcept]
Get the padding of the deconvolution.
Default: (0, 0)
See also
setPadding()
Deprecated Superseded by getPaddingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
266
Class Documentation
9.52.3.9
getPaddingMode()
PaddingMode nvinfer1::IDeconvolutionLayer::getPaddingMode ( ) const [inline], [noexcept]
Get the padding mode.
Default: kEXPLICIT ROUND DOWN
See also
setPaddingMode()
9.52.3.10 getPaddingNd()
Dims nvinfer1::IDeconvolutionLayer::getPaddingNd ( ) const [inline], [noexcept]
Get the multi-dimension padding of the deconvolution.
If the padding is asymmetric, the pre-padding is returned.
See also
setPaddingNd()
9.52.3.11 getPostPadding()
Dims nvinfer1::IDeconvolutionLayer::getPostPadding ( ) const [inline], [noexcept]
Get the padding.
See also
setPostPadding()
9.52.3.12 getPrePadding()
Dims nvinfer1::IDeconvolutionLayer::getPrePadding ( ) const [inline], [noexcept]
Get the pre-padding.
See also
setPrePadding()
9.52 nvinfer1::IDeconvolutionLayer Class Reference
267
9.52.3.13 getStride()
TRT DEPRECATED DimsHW nvinfer1::IDeconvolutionLayer::getStride ( ) const [inline], [noexcept]
Get the stride of the deconvolution.
Default: (1,1)
Deprecated Superseded by getStrideNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.52.3.14 getStrideNd()
Dims nvinfer1::IDeconvolutionLayer::getStrideNd ( ) const [inline], [noexcept]
Get the multi-dimension stride of the deconvolution.
See also
setStrideNd()
9.52.3.15 setBiasWeights()
void nvinfer1::IDeconvolutionLayer::setBiasWeights (
Weights weights ) [inline], [noexcept]
Set the bias weights for the deconvolution.
Bias is optional. To omit bias, set the count value of the weights structure to zero.
The bias is applied per-feature-map, so the number of weights (if non-zero) must be equal to the number of output
feature maps.
See also
getBiasWeights()
268
Class Documentation
9.52.3.16 setDilationNd()
void nvinfer1::IDeconvolutionLayer::setDilationNd (
Dims dilation ) [inline], [noexcept]
Set the multi-dimension dilation of the deconvolution.
Default: (1, 1, ..., 1)
See also
getDilationNd()
9.52.3.17 setInput()
void nvinfer1::ILayer::setInput (
int32 t index,
ITensor & tensor ) [inline], [noexcept]
Append or replace an input of this layer with a specific tensor.
Parameters
index
the index of the input to modify.
tensor
the new input tensor
Input 0 is the input activation tensor. Input 1 is the kernel tensor. If used, the kernel weights parameter must be set to
empty weights. Input 2 is the bias tensor. If used, the bias parameter must be set to empty weights.
See also
getKernelWeights(), setKernelWeights(), getBiasWeights(), setBiasWeights()
9.52.3.18 setKernelSize()
TRT DEPRECATED void nvinfer1::IDeconvolutionLayer::setKernelSize (
DimsHW kernelSize ) [inline], [noexcept]
Set the HW kernel size of the convolution.
If executing this layer on DLA, both height and width of kernel size must be in the range [1,32], or the combinations
of [64, 96, 128] in one dimension and 1 in the other dimensions, i.e. [1x64] or [64x1] are valid, but not [64x64].
9.52 nvinfer1::IDeconvolutionLayer Class Reference
269
See also
getKernelSize()
Deprecated Superseded by setKernelSizeNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.52.3.19 setKernelSizeNd()
void nvinfer1::IDeconvolutionLayer::setKernelSizeNd (
Dims kernelSize ) [inline], [noexcept]
Set the multi-dimension kernel size of the deconvolution.
If executing this layer on DLA, there are ttwo restrictions: 1) Only 2D Kernel is supported. 2) Kernel height and width
must be in the range [1,32] or the combinations of [64, 96, 128] in one dimension and 1 in the other dimensions, i.e.
[1x64] or [64x1] are valid, but not [64x64].
See also
getKernelSizeNd() setKernelSize() getKernelSize()
9.52.3.20 setKernelWeights()
void nvinfer1::IDeconvolutionLayer::setKernelWeights (
Weights weights ) [inline], [noexcept]
Set the kernel weights for the deconvolution.
The weights are specified as a contiguous array in CKRS order, where C the number of input channels, K the number
of output feature maps, and R and S are the height and width of the filter.
See also
getWeights()
270
Class Documentation
9.52.3.21 setNbGroups()
void nvinfer1::IDeconvolutionLayer::setNbGroups (
int32 t nbGroups ) [inline], [noexcept]
Set the number of groups for a deconvolution.
The input tensor channels are divided into nbGroups groups, and a deconvolution is executed for each group, using
a filter per group. The results of the group convolutions are concatenated to form the output.
If executing this layer on DLA, nbGroups must be one
Note
When using groups in int8 mode, the size of the groups (i.e. the channel count divided by the group count) must
be a multiple of 4 for both input and output.
Default: 1
See also
getNbGroups()
9.52.3.22 setNbOutputMaps()
void nvinfer1::IDeconvolutionLayer::setNbOutputMaps (
int32 t nbOutputMaps ) [inline], [noexcept]
Set the number of output feature maps for the deconvolution.
If executing this layer on DLA, the number of output maps must be in the range [1,8192].
See also
getNbOutputMaps()
9.52 nvinfer1::IDeconvolutionLayer Class Reference
271
9.52.3.23 setPadding()
TRT DEPRECATED void nvinfer1::IDeconvolutionLayer::setPadding (
DimsHW padding ) [inline], [noexcept]
Set the padding of the deconvolution.
The output will be trimmed by this number of elements on each side in the height and width directions. In other words,
it resembles the inverse of a convolution layer with this padding size. Padding is symmetric, and negative padding is
not supported.
Default: (0,0)
If executing this layer on DLA, both height and width of padding must be 0.
See also
getPadding()
Deprecated Superseded by setPaddingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.52.3.24 setPaddingMode()
void nvinfer1::IDeconvolutionLayer::setPaddingMode (
PaddingMode paddingMode ) [inline], [noexcept]
Set the padding mode.
Padding mode takes precedence if both setPaddingMode and setPre/PostPadding are used.
Default: kEXPLICIT ROUND DOWN
See also
getPaddingMode()
9.52.3.25 setPaddingNd()
void nvinfer1::IDeconvolutionLayer::setPaddingNd (
Dims padding ) [inline], [noexcept]
Set the multi-dimension padding of the deconvolution.
The output will be trimmed by this number of elements on both sides of every dimension. In other words, it resembles
the inverse of a convolution layer with this padding size. Padding is symmetric, and negative padding is not supported.
Default: (0, 0, ..., 0)
If executing this layer on DLA, padding must be 0.
See also
getPaddingNd() setPadding() getPadding()
272
Class Documentation
9.52.3.26 setPostPadding()
void nvinfer1::IDeconvolutionLayer::setPostPadding (
Dims padding ) [inline], [noexcept]
Set the multi-dimension post-padding of the deconvolution.
The output will be trimmed by this number of elements on the end of every dimension. In other words, it resembles
the inverse of a convolution layer with this padding size. Negative padding is not supported.
Default: (0, 0, ..., 0)
If executing this layer on DLA, padding must be 0.
See also
getPostPadding()
9.52.3.27 setPrePadding()
void nvinfer1::IDeconvolutionLayer::setPrePadding (
Dims padding ) [inline], [noexcept]
Set the multi-dimension pre-padding of the deconvolution.
The output will be trimmed by this number of elements on the start of every dimension. In other words, it resembles
the inverse of a convolution layer with this padding size. Negative padding is not supported.
Default: (0, 0, ..., 0)
If executing this layer on DLA, padding must be 0.
See also
getPrePadding()
9.52.3.28 setStride()
TRT DEPRECATED void nvinfer1::IDeconvolutionLayer::setStride (
DimsHW stride ) [inline], [noexcept]
Set the stride of the deconvolution.
If executing this layer on DLA, there is one restriction: 1) Stride height and width must be in the range [1,32] or the
combinations of [64, 96, 128] in one dimension and 1 in the other dimensions, i.e. [1x64] or [64x1] are valid, but not
[64x64].
See also
getStride()
Deprecated Superseded by setStrideNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.53 nvinfer1::IDequantizeLayer Class Reference
273
9.52.3.29 setStrideNd()
void nvinfer1::IDeconvolutionLayer::setStrideNd (
Dims stride ) [inline], [noexcept]
Set the multi-dimension stride of the deconvolution.
Default: (1, 1, ..., 1)
If executing this layer on DLA, there are two restrictions: 1) Only 2D Stride is supported. 2) Stride height and width
must be in the range [1,32] or the combinations of [64, 96, 128] in one dimension and 1 in the other dimensions, i.e.
[1x64] or [64x1] are valid, but not [64x64].
See also
getStrideNd() setStride() getStride()
9.52.4
Member Data Documentation
9.52.4.1
mImpl
apiv::VDeconvolutionLayer∗ nvinfer1::IDeconvolutionLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.53
nvinfer1::IDequantizeLayer Class Reference
A Dequantize layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IDequantizeLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IDequantizeLayer
274
Class Documentation
Public Member Functions
• int32 t getAxis () const noexcept
Get the quantization axis.
• void setAxis (int32 t axis) noexcept
Set the quantization axis.
Protected Member Functions
• virtual ∼IDequantizeLayer () noexcept=default
Protected Attributes
• apiv::VDequantizeLayer ∗ mImpl
9.53.1
Detailed Description
A Dequantize layer in a network definition.
This layer accepts a signed 8-bit integer input tensor, and uses the configured scale and zeroPt inputs to dequantize the
input according to: output = (input - zeroPt) ∗ scale
The first input (index 0) is the tensor to be quantized. The second (index 1) and third (index 2) are the scale and zero
point respectively. Each of scale and zeroPt must be either a scalar, or a 1D tensor.
The zeroPt tensor is optional, and if not set, will be assumed to be zero. Its data type must be DataType::kINT8.
zeroPt must only contain zero-valued coefficients, because only symmetric quantization is supported. The scale
value must be either a scalar for per-tensor quantization, or a 1D tensor for per-channel quantization. All scale
coefficients must have positive values. The size of the 1-D scale tensor must match the size of the quantization axis.
The size of the scale must match the size of the zeroPt.
The subgraph which terminates with the scale tensor must be a build-time constant. The same restrictions apply to
the zeroPt. The output type, if constrained, must be constrained to DataType::kFLOAT or DataType::kHALF. The
input type, if constrained, must be constrained to DataType::kINT8. The output size is the same as the input size. The
quantization axis is in reference to the input tensor's dimensions.
IDequantizeLayer only supports DataType::kINT8 precision and will default to this precision during instantiation.
IDequantizeLayer only supports DataType::kFLOAT or DataType::kHALF output.
As an example of the operation of this layer, imagine a 4D NCHW activation input which can be quantized using a
single scale coefficient (referred to as per-tensor quantization): For each n in N: For each c in C: For each h in H: For
each w in W: output[n,c,h,w] = (input[n,c,h,w] - zeroPt) ∗ scale
Per-channel dequantization is supported only for input that is rooted at an IConstantLayer (i.e. weights). Activations
cannot be quantized per-channel. As an example of per-channel operation, imagine a 4D KCRS weights input and K
(dimension 0) as the quantization axis. The scale is an array of coefficients, which is the same size as the quantization
axis. For each k in K: For each c in C: For each r in R: For each s in S: output[k,c,r,s] = (input[k,c,r,s] - zeroPt[k])
∗ scale[k]
Note
Only symmetric quantization is supported.
Currently the only allowed build-time constant scale and zeroPt subgraphs are:
1. Constant -> Quantize
2. Constant -> Cast -> Quantize
9.53 nvinfer1::IDequantizeLayer Class Reference
275
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.53.2
Constructor & Destructor Documentation
9.53.2.1
∼IDequantizeLayer()
virtual nvinfer1::IDequantizeLayer::∼IDequantizeLayer ( ) [protected], [virtual], [default],
[noexcept]
9.53.3
Member Function Documentation
9.53.3.1
getAxis()
int32 t nvinfer1::IDequantizeLayer::getAxis ( ) const [inline], [noexcept]
Get the quantization axis.
Returns
axis parameter set by setAxis(). The return value is the index of the quantization axis in the input tensor's
dimensions. A value of -1 indicates per-tensor quantization. The default value is -1.
9.53.3.2
setAxis()
void nvinfer1::IDequantizeLayer::setAxis (
int32 t axis ) [inline], [noexcept]
Set the quantization axis.
Set the index of the quantization axis (with reference to the input tensor's dimensions). The axis must be a valid axis
if the scale tensor has more than one coefficient. The axis value will be ignored if the scale tensor has exactly one
coefficient (per-tensor quantization).
9.53.4
Member Data Documentation
276
Class Documentation
9.53.4.1
mImpl
apiv::VDequantizeLayer∗ nvinfer1::IDequantizeLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.54
nvinfer1::IDimensionExpr Class Reference
#include <NvInferRuntime.h>
Inheritance diagram for nvinfer1::IDimensionExpr:
nvinfer1::INoCopy
nvinfer1::IDimensionExpr
Public Member Functions
• bool isConstant () const noexcept
Return true if expression is a build-time constant.
• int32 t getConstantValue () const noexcept
Protected Member Functions
• virtual ∼IDimensionExpr () noexcept=default
Protected Attributes
• apiv::VDimensionExpr ∗ mImpl
9.54.1
Detailed Description
An IDimensionExpr represents an integer expression constructed from constants, input dimensions, and binary opera-
tions. These expressions are can be used in overrides of IPluginV2DynamicExt::getOutputDimensions to define output
dimensions in terms of input dimensions.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.54 nvinfer1::IDimensionExpr Class Reference
277
See also
DimensionOperation, IPluginV2DynamicExt::getOutputDimensions
9.54.2
Constructor & Destructor Documentation
9.54.2.1
∼IDimensionExpr()
virtual nvinfer1::IDimensionExpr::∼IDimensionExpr ( ) [protected], [virtual], [default], [noexcept]
9.54.3
Member Function Documentation
9.54.3.1
getConstantValue()
int32 t nvinfer1::IDimensionExpr::getConstantValue ( ) const [inline], [noexcept]
If isConstant(), returns value of the constant. If !isConstant(), return std::numeric limits<int32 t>::min().
9.54.3.2
isConstant()
bool nvinfer1::IDimensionExpr::isConstant ( ) const [inline], [noexcept]
Return true if expression is a build-time constant.
9.54.4
Member Data Documentation
9.54.4.1
mImpl
apiv::VDimensionExpr∗ nvinfer1::IDimensionExpr::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInferRuntime.h
278
Class Documentation
9.55
nvinfer1::IEinsumLayer Class Reference
An Einsum layer in a network.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IEinsumLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IEinsumLayer
Public Member Functions
• bool setEquation (char const ∗equation) noexcept
Set the equation. The equation is a comma-separated list of subscript labels, where each label refers to a dimension of
the corresponding tensor.
• char const ∗ getEquation () const noexcept
Return the equation.
Protected Member Functions
• virtual ∼IEinsumLayer () noexcept=default
Protected Attributes
• apiv::VEinsumLayer ∗ mImpl
9.55.1
Detailed Description
An Einsum layer in a network.
This layer implements a summation over the elements of the inputs along dimensions specified by the equation pa-
rameter, based on the Einstein summation convention. The layer can have one or more inputs of rank >= 0. All the
inputs must have type DataType::kFLOAT or DataType::kHALF, not necessarily the same. There is one output of type
DataType::kFLOAT. The shape of the output tensor is determined by the equation.
The equation specifies ASCII lower-case letters for each dimension in the inputs in the same order as the dimensions,
separated by comma for each input. The dimensions labeled with the same subscript must match or be broadcastable.
Repeated subscript labels in one input take the diagonal. Repeating a label across multiple inputs means that those
axes will be multiplied. Omitting a label from the output means values along those axes will be summed. In implicit
mode, the indices which appear once in the expression will be part of the output in increasing alphabetical order. In
explicit mode, the output can be controlled by specifying output subscript labels by adding an arrow (‘->’) followed
by subscripts for the output. For example, “ij,jk->ik” is equivalent to “ij,jk”. Ellipsis (‘...’) can be used in place of
subscripts to broadcast the dimensions. See the TensorRT Developer Guide for more details on equation syntax.
Many common operations can be expressed using the Einsum equation. For example: Matrix Transpose: ij->ji Sum:
ij-> Matrix-Matrix Multiplication: ik,kj->ij Dot Product: i,i-> Matrix-Vector Multiplication: ik,k->i Batch Matrix
Multiplication: ijk,ikl->ijl Batch Diagonal: ...ii->...i
9.55 nvinfer1::IEinsumLayer Class Reference
279
Note
TensorRT does not support ellipsis, diagonal operations or more than two inputs for Einsum.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.55.2
Constructor & Destructor Documentation
9.55.2.1
∼IEinsumLayer()
virtual nvinfer1::IEinsumLayer::∼IEinsumLayer ( ) [protected], [virtual], [default], [noexcept]
9.55.3
Member Function Documentation
9.55.3.1
getEquation()
char const ∗ nvinfer1::IEinsumLayer::getEquation ( ) const [inline], [noexcept]
Return the equation.
See also
setEquation()
9.55.3.2
setEquation()
bool nvinfer1::IEinsumLayer::setEquation (
char const ∗ equation ) [inline], [noexcept]
Set the equation. The equation is a comma-separated list of subscript labels, where each label refers to a dimension of
the corresponding tensor.
Returns
true if the equation was syntactically valid and set successfully, false otherwise.
See also
setEquation()
280
Class Documentation
9.55.4
Member Data Documentation
9.55.4.1
mImpl
apiv::VEinsumLayer∗ nvinfer1::IEinsumLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.56
nvinfer1::IElementWiseLayer Class Reference
A elementwise layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IElementWiseLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IElementWiseLayer
Public Member Functions
• void setOperation (ElementWiseOperation op) noexcept
Set the binary operation for the layer.
• ElementWiseOperation getOperation () const noexcept
Get the binary operation for the layer.
Protected Member Functions
• virtual ∼IElementWiseLayer () noexcept=default
Protected Attributes
• apiv::VElementWiseLayer ∗ mImpl
9.56 nvinfer1::IElementWiseLayer Class Reference
281
9.56.1
Detailed Description
A elementwise layer in a network definition.
This layer applies a per-element binary operation between corresponding elements of two tensors.
The input tensors must have the same rank. For each dimension, their lengths must match, or one of them must be one.
In the latter case, the tensor is broadcast along that axis.
The output tensor has the same rank as the inputs. For each output dimension, its length is equal to the lengths of the
corresponding input dimensions if they match, otherwise it is equal to the length that is not one.
Warning
When running this layer on the DLA with Int8 data type, the dynamic ranges of two input tensors shall be
equal. If the dynamic ranges are generated using calibrator, the largest value shall be used.
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.56.2
Constructor & Destructor Documentation
9.56.2.1
∼IElementWiseLayer()
virtual nvinfer1::IElementWiseLayer::∼IElementWiseLayer ( ) [protected], [virtual], [default],
[noexcept]
9.56.3
Member Function Documentation
9.56.3.1
getOperation()
ElementWiseOperation nvinfer1::IElementWiseLayer::getOperation ( ) const [inline], [noexcept]
Get the binary operation for the layer.
See also
setOperation(), ElementWiseOperation
setBiasWeights()
282
Class Documentation
9.56.3.2
setOperation()
void nvinfer1::IElementWiseLayer::setOperation (
ElementWiseOperation op ) [inline], [noexcept]
Set the binary operation for the layer.
DLA supports only kSUM, kPROD, kMAX, kMIN, and kSUB.
See also
getOperation(), ElementWiseOperation
getBiasWeights()
9.56.4
Member Data Documentation
9.56.4.1
mImpl
apiv::VElementWiseLayer∗ nvinfer1::IElementWiseLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.57
nvinfer1::IEngineInspector Class Reference
An engine inspector which prints out the layer information of an engine or an execution context.
#include <NvInferRuntime.h>
Inheritance diagram for nvinfer1::IEngineInspector:
nvinfer1::INoCopy
nvinfer1::IEngineInspector
9.57 nvinfer1::IEngineInspector Class Reference
283
Public Member Functions
• virtual ∼IEngineInspector () noexcept=default
• bool setExecutionContext (IExecutionContext const ∗context) noexcept
Set an execution context as the inspection source.
• IExecutionContext const ∗ getExecutionContext () const noexcept
Get the context currently being inspected.
• char const ∗ getLayerInformation (int32 t layerIndex, LayerInformationFormat format) const noexcept
Get a string describing the information about a specific layer in the current engine or the execution context.
• char const ∗ getEngineInformation (LayerInformationFormat format) const noexcept
Get a string describing the information about all the layers in the current engine or the execution context.
• void setErrorRecorder (IErrorRecorder ∗recorder) noexcept
Set the ErrorRecorder for this interface.
• IErrorRecorder ∗ getErrorRecorder () const noexcept
Get the ErrorRecorder assigned to this interface.
Protected Attributes
• apiv::VEngineInspector ∗ mImpl
Additional Inherited Members
9.57.1
Detailed Description
An engine inspector which prints out the layer information of an engine or an execution context.
The amount of printed information depends on the profiling verbosity setting of the builder config when the engine is
built:
• ProfilingVerbosity::kLAYER NAMES ONLY: only layer names will be printed.
• ProfilingVerbosity::kNONE: no layer information will be printed.
• ProfilingVerbosity::kDETAILED: layer names and layer parameters will be printed.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
See also
ProfilingVerbosity, IEngineInspector
9.57.2
Constructor & Destructor Documentation
284
Class Documentation
9.57.2.1
∼IEngineInspector()
virtual nvinfer1::IEngineInspector::∼IEngineInspector ( ) [virtual], [default], [noexcept]
9.57.3
Member Function Documentation
9.57.3.1
getEngineInformation()
char const ∗ nvinfer1::IEngineInspector::getEngineInformation (
LayerInformationFormat format ) const [inline], [noexcept]
Get a string describing the information about all the layers in the current engine or the execution context.
Parameters
format
the format the layer information should be printed in.
Returns
A null-terminated C-style string describing the information about all the layers in the current engine or the
execution context.
Warning
The content of the returned string may change when another execution context has been set, or when another
getLayerInformation() or getEngineInformation() has been called.
In a multi-threaded environment, this function must be protected from other threads changing the inspection
source. If the inspection source changes, the data that is being pointed to can change. Copy the string to
another buffer before releasing the lock in order to guarantee consistency.
See also
LayerInformationFormat
9.57.3.2
getErrorRecorder()
IErrorRecorder ∗ nvinfer1::IEngineInspector::getErrorRecorder ( ) const [inline], [noexcept]
Get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A nullptr will be returned if an error handler has not
been set.
9.57 nvinfer1::IEngineInspector Class Reference
285
Returns
A pointer to the IErrorRecorder object that has been registered.
See also
setErrorRecorder()
9.57.3.3
getExecutionContext()
IExecutionContext const ∗ nvinfer1::IEngineInspector::getExecutionContext ( ) const [inline],
[noexcept]
Get the context currently being inspected.
Returns
The pointer to the context currently being inspected.
See also
setExecutionContext()
9.57.3.4
getLayerInformation()
char const ∗ nvinfer1::IEngineInspector::getLayerInformation (
int32 t layerIndex,
LayerInformationFormat format ) const [inline], [noexcept]
Get a string describing the information about a specific layer in the current engine or the execution context.
Parameters
layerIndex
the index of the layer. It must lie in range [0, engine.getNbLayers()).
format
the format the layer information should be printed in.
Returns
A null-terminated C-style string describing the information about a specific layer in the current engine or the
execution context.
286
Class Documentation
Warning
The content of the returned string may change when another execution context has been set, or when another
getLayerInformation() or getEngineInformation() has been called.
In a multi-threaded environment, this function must be protected from other threads changing the inspection
source. If the inspection source changes, the data that is being pointed to can change. Copy the string to
another buffer before releasing the lock in order to guarantee consistency.
See also
LayerInformationFormat
9.57.3.5
setErrorRecorder()
void nvinfer1::IEngineInspector::setErrorRecorder (
IErrorRecorder ∗ recorder ) [inline], [noexcept]
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function
will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder
with the interface, resulting in a call to decRefCount if a recorder has been registered.
If an error recorder is not set, messages will be sent to the global log stream.
Parameters
recorder
The error recorder to register with this interface.
See also
getErrorRecorder()
9.57.3.6
setExecutionContext()
bool nvinfer1::IEngineInspector::setExecutionContext (
IExecutionContext const ∗ context ) [inline], [noexcept]
Set an execution context as the inspection source.
Setting the execution context and specifying all the input shapes allows the inspector to calculate concrete dimensions
for any dynamic shapes and display their format information. Otherwise, values dependent on input shapes will be
displayed as -1 and format information will not be shown.
Passing nullptr will remove any association with an execution context.
9.58 nvinfer1::IErrorRecorder Class Reference
287
Returns
Whether the action succeeds.
9.57.4
Member Data Documentation
9.57.4.1
mImpl
apiv::VEngineInspector∗ nvinfer1::IEngineInspector::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInferRuntime.h
9.58
nvinfer1::IErrorRecorder Class Reference
Reference counted application-implemented error reporting interface for TensorRT objects.
#include <NvInferRuntimeBase.h>
Public Types
• using ErrorDesc = char const ∗
• using RefCount = int32 t
Public Member Functions
• IErrorRecorder ()=default
• virtual ∼IErrorRecorder () noexcept=default
• virtual int32 t getNbErrors () const noexcept=0
Return the number of errors.
• virtual ErrorCode getErrorCode (int32 t errorIdx) const noexcept=0
Returns the ErrorCode enumeration.
• virtual ErrorDesc getErrorDesc (int32 t errorIdx) const noexcept=0
Returns a null-terminated C-style string description of the error.
• virtual bool hasOverflowed () const noexcept=0
Determine if the error stack has overflowed.
• virtual void clear () noexcept=0
Clear the error stack on the error recorder.
• virtual bool reportError (ErrorCode val, ErrorDesc desc) noexcept=0
Report an error to the error recorder with the corresponding enum and description.
• virtual RefCount incRefCount () noexcept=0
Increments the refcount for the current ErrorRecorder.
• virtual RefCount decRefCount () noexcept=0
Decrements the refcount for the current ErrorRecorder.
288
Class Documentation
Static Public Attributes
• static constexpr size t kMAX DESC LENGTH {127U}
9.58.1
Detailed Description
Reference counted application-implemented error reporting interface for TensorRT objects.
The error reporting mechanism is a user defined object that interacts with the internal state of the object that it is
assigned to in order to determine information about abnormalities in execution. The error recorder gets both an error
enum that is more descriptive than pass/fail and also a string description that gives more detail on the exact failure
modes. In the safety context, the error strings are all limited to 1024 characters in length.
The ErrorRecorder gets passed along to any class that is created from another class that has an ErrorRecorder assigned
to it. For example, assigning an ErrorRecorder to an IBuilder allows all INetwork's, ILayer's, and ITensor's to use the
same error recorder. For functions that have their own ErrorRecorder accessor functions. This allows registering a
different error recorder or de-registering of the error recorder for that specific object.
The ErrorRecorder object implementation must be thread safe. All locking and synchronization is pushed to the inter-
face implementation and TensorRT does not hold any synchronization primitives when calling the interface functions.
The lifetime of the ErrorRecorder object must exceed the lifetime of all TensorRT objects that use it.
9.58.2
Member Typedef Documentation
9.58.2.1
ErrorDesc
using nvinfer1::IErrorRecorder::ErrorDesc = char const∗
A typedef of a C-style string for reporting error descriptions.
9.58.2.2
RefCount
using nvinfer1::IErrorRecorder::RefCount = int32 t
A typedef of a 32bit integer for reference counting.
9.58.3
Constructor & Destructor Documentation
9.58 nvinfer1::IErrorRecorder Class Reference
289
9.58.3.1
IErrorRecorder()
nvinfer1::IErrorRecorder::IErrorRecorder ( ) [default]
9.58.3.2
∼IErrorRecorder()
virtual nvinfer1::IErrorRecorder::∼IErrorRecorder ( ) [virtual], [default], [noexcept]
9.58.4
Member Function Documentation
9.58.4.1
clear()
virtual void nvinfer1::IErrorRecorder::clear ( ) [pure virtual], [noexcept]
Clear the error stack on the error recorder.
Removes all the tracked errors by the error recorder. This function must guarantee that after this function is called, and
as long as no error occurs, the next call to getNbErrors will return zero.
See also
getNbErrors
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
multiple execution contexts are used during runtime.
290
Class Documentation
9.58.4.2
decRefCount()
virtual RefCount nvinfer1::IErrorRecorder::decRefCount ( ) [pure virtual], [noexcept]
Decrements the refcount for the current ErrorRecorder.
Decrements the reference count for the object by one and returns the current value. This reference count allows
the application to know that an object inside of TensorRT has taken a reference to the ErrorRecorder. TensorRT
guarantees that every call to IErrorRecorder::decRefCount will be preceded by a call to IErrorRecorder::incRefCount.
It is undefined behavior to destruct the ErrorRecorder when incRefCount has been called without a corresponding
decRefCount.
Returns
The reference counted value after the decrement completes.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
multiple execution contexts are used during runtime.
9.58.4.3
getErrorCode()
virtual ErrorCode nvinfer1::IErrorRecorder::getErrorCode (
int32 t errorIdx ) const [pure virtual], [noexcept]
Returns the ErrorCode enumeration.
Parameters
errorIdx
A 32-bit integer that indexes into the error array.
The errorIdx specifies what error code from 0 to getNbErrors()-1 that the application wants to analyze and return the
error code enum.
Returns
Returns the enum corresponding to errorIdx.
9.58 nvinfer1::IErrorRecorder Class Reference
291
See also
getErrorDesc, ErrorCode
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
multiple execution contexts are used during runtime.
9.58.4.4
getErrorDesc()
virtual ErrorDesc nvinfer1::IErrorRecorder::getErrorDesc (
int32 t errorIdx ) const [pure virtual], [noexcept]
Returns a null-terminated C-style string description of the error.
Parameters
errorIdx
A 32-bit integer that indexes into the error array.
For the error specified by the idx value, return the string description of the error. The error string is a null-terminated
C-style string. In the safety context there is a constant length requirement to remove any dynamic memory allocations
and the error message may be truncated. The format of the string is ”<EnumAsStr> - <Description>”.
Returns
Returns a string representation of the error along with a description of the error.
See also
getErrorCode
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
multiple execution contexts are used during runtime.
292
Class Documentation
9.58.4.5
getNbErrors()
virtual int32 t nvinfer1::IErrorRecorder::getNbErrors ( ) const [pure virtual], [noexcept]
Return the number of errors.
Determines the number of errors that occurred between the current point in execution and the last time that the clear()
was executed. Due to the possibility of asynchronous errors occuring, a TensorRT API can return correct results, but
still register errors with the Error Recorder. The value of getNbErrors must monotonically increases until clear() is
called.
Returns
Returns the number of errors detected, or 0 if there are no errors.
See also
clear
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
multiple execution contexts are used during runtime.
9.58.4.6
hasOverflowed()
virtual bool nvinfer1::IErrorRecorder::hasOverflowed ( ) const [pure virtual], [noexcept]
Determine if the error stack has overflowed.
In the case when the number of errors is large, this function is used to query if one or more errors have been dropped
due to lack of storage capacity. This is especially important in the automotive safety case where the internal error
handling mechanisms cannot allocate memory.
Returns
true if errors have been dropped due to overflowing the error stack.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
multiple execution contexts are used during runtime.
9.58 nvinfer1::IErrorRecorder Class Reference
293
9.58.4.7
incRefCount()
virtual RefCount nvinfer1::IErrorRecorder::incRefCount ( ) [pure virtual], [noexcept]
Increments the refcount for the current ErrorRecorder.
Increments the reference count for the object by one and returns the current value. This reference count allows the
application to know that an object inside of TensorRT has taken a reference to the ErrorRecorder. TensorRT guarantees
that every call to IErrorRecorder::incRefCount will be paired with a call to IErrorRecorder::decRefCount when the
reference is released. It is undefined behavior to destruct the ErrorRecorder when incRefCount has been called without
a corresponding decRefCount.
Returns
The reference counted value after the increment completes.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
multiple execution contexts are used during runtime.
9.58.4.8
reportError()
virtual bool nvinfer1::IErrorRecorder::reportError (
ErrorCode val,
ErrorDesc desc ) [pure virtual], [noexcept]
Report an error to the error recorder with the corresponding enum and description.
Parameters
val
The error code enum that is being reported.
desc
The string description of the error.
Report an error to the user that has a given value and human readable description. The function returns false if
processing can continue, which implies that the reported error is not fatal. This does not guarantee that processing
continues, but provides a hint to TensorRT. The desc C-string data is only valid during the call to reportError and may
be immediately deallocated by the caller when reportError returns. The implementation must not store the desc pointer
in the ErrorRecorder object or otherwise access the data from desc after reportError returns.
294
Class Documentation
Returns
True if the error is determined to be fatal and processing of the current function must end.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
multiple execution contexts are used during runtime.
9.58.5
Member Data Documentation
9.58.5.1
kMAX DESC LENGTH
constexpr size t nvinfer1::IErrorRecorder::kMAX DESC LENGTH {127U} [static], [constexpr]
The length limit for an error description, excluding the '\0' string terminator.
The documentation for this class was generated from the following file:
• NvInferRuntimeBase.h
9.59
nvinfer1::IExecutionContext Class Reference
Context for executing inference using an engine, with functionally unsafe features.
#include <NvInferRuntime.h>
Inheritance diagram for nvinfer1::IExecutionContext:
nvinfer1::INoCopy
nvinfer1::IExecutionContext
9.59 nvinfer1::IExecutionContext Class Reference
295
Public Member Functions
•
virtual ∼IExecutionContext () noexcept=default
•
TRT DEPRECATED bool execute (int32 t batchSize, void ∗const ∗bindings) noexcept
Synchronously execute inference on a batch.
•
TRT DEPRECATED bool enqueue (int32 t batchSize, void ∗const ∗bindings, cudaStream t stream, cuda←↩
Event t ∗inputConsumed) noexcept
Enqueue inference of a batch on a stream.
•
void setDebugSync (bool sync) noexcept
Set the debug sync flag.
•
bool getDebugSync () const noexcept
Get the debug sync flag.
•
void setProfiler (IProfiler ∗profiler) noexcept
Set the profiler.
•
IProfiler ∗ getProfiler () const noexcept
Get the profiler.
•
ICudaEngine const & getEngine () const noexcept
Get the associated engine.
•
TRT DEPRECATED void destroy () noexcept
Destroy this object.
•
void setName (char const ∗name) noexcept
Set the name of the execution context.
•
char const ∗ getName () const noexcept
Return the name of the execution context.
•
void setDeviceMemory (void ∗memory) noexcept
Set the device memory for use by this execution context.
•
TRT DEPRECATED Dims getStrides (int32 t bindingIndex) const noexcept
Return the strides of the buffer for the given binding.
•
Dims getTensorStrides (char const ∗tensorName) const noexcept
Return the strides of the buffer for the given tensor name.
•
TRT DEPRECATED bool setOptimizationProfile (int32 t profileIndex) noexcept
Select an optimization profile for the current context.
•
int32 t getOptimizationProfile () const noexcept
Get the index of the currently selected optimization profile.
•
TRT DEPRECATED bool setBindingDimensions (int32 t bindingIndex, Dims dimensions) noexcept
Set the dynamic dimensions of an input binding.
•
bool setInputShape (char const ∗tensorName, Dims const &dims) noexcept
Set shape of given input.
•
TRT DEPRECATED Dims getBindingDimensions (int32 t bindingIndex) const noexcept
Get the dynamic dimensions of a binding.
•
Dims getTensorShape (char const ∗tensorName) const noexcept
Return the shape of the given input or output.
•
TRT DEPRECATED bool setInputShapeBinding (int32 t bindingIndex, int32 t const ∗data) noexcept
Set values of input tensor required by shape calculations.
•
TRT DEPRECATED bool getShapeBinding (int32 t bindingIndex, int32 t ∗data) const noexcept
Get values of an input tensor required for shape calculations or an output tensor produced by shape calculations.
•
bool allInputDimensionsSpecified () const noexcept
296
Class Documentation
Whether all dynamic dimensions of input tensors have been specified.
•
bool allInputShapesSpecified () const noexcept
Whether all input shape bindings have been specified.
•
void setErrorRecorder (IErrorRecorder ∗recorder) noexcept
Set the ErrorRecorder for this interface.
•
IErrorRecorder ∗ getErrorRecorder () const noexcept
Get the ErrorRecorder assigned to this interface.
•
bool executeV2 (void ∗const ∗bindings) noexcept
Synchronously execute inference a network.
•
TRT DEPRECATED bool enqueueV2 (void ∗const ∗bindings, cudaStream t stream, cudaEvent t ∗input←↩
Consumed) noexcept
Enqueue inference on a stream.
•
bool setOptimizationProfileAsync (int32 t profileIndex, cudaStream t stream) noexcept
Select an optimization profile for the current context with async semantics.
•
void setEnqueueEmitsProfile (bool enqueueEmitsProfile) noexcept
Set whether enqueue emits layer timing to the profiler.
•
bool getEnqueueEmitsProfile () const noexcept
Get the enqueueEmitsProfile state.
•
bool reportToProfiler () const noexcept
Calculate layer timing info for the current optimization profile in IExecutionContext and update the profiler after one
iteration of inference launch.
•
bool setTensorAddress (char const ∗tensorName, void ∗data) noexcept
Set memory address for given input or output tensor.
•
void const ∗ getTensorAddress (char const ∗tensorName) const noexcept
Get memory address bound to given input or output tensor, or nullptr if the provided name does not map to an input or
output tensor.
•
bool setInputTensorAddress (char const ∗tensorName, void const ∗data) noexcept
Set memory address for given input.
•
void ∗ getOutputTensorAddress (char const ∗tensorName) const noexcept
Get memory address for given output.
•
int32 t inferShapes (int32 t nbMaxNames, char const ∗∗tensorNames) noexcept
Run shape calculations.
•
bool setInputConsumedEvent (cudaEvent t event) noexcept
Mark input as consumed.
•
cudaEvent t getInputConsumedEvent () const noexcept
The event associated with consuming the input.
•
bool setOutputAllocator (char const ∗tensorName, IOutputAllocator ∗outputAllocator) noexcept
Set output allocator to use for output tensor of given name. Pass nullptr to outputAllocator to unset. The allocator is
called by enqueueV3().
•
IOutputAllocator ∗ getOutputAllocator (char const ∗tensorName) const noexcept
Get output allocator associated with output tensor of given name, or nullptr if the provided name does not map to an
output tensor.
•
int64 t getMaxOutputSize (char const ∗tensorName) const noexcept
Get upper bound on an output tensor's size, in bytes, based on the current optimization profile and input dimensions.
•
bool setTemporaryStorageAllocator (IGpuAllocator ∗allocator) noexcept
Specify allocator to use for internal temporary storage.
•
IGpuAllocator ∗ getTemporaryStorageAllocator () const noexcept
Get allocator set by setTemporaryStorageAllocator.
9.59 nvinfer1::IExecutionContext Class Reference
297
• bool enqueueV3 (cudaStream t stream) noexcept
Enqueue inference on a stream.
• void setPersistentCacheLimit (size t size) noexcept
Set the maximum size for persistent cache usage.
• size t getPersistentCacheLimit () const noexcept
Get the maximum size for persistent cache usage.
• bool setNvtxVerbosity (ProfilingVerbosity verbosity) noexcept
Set the verbosity of the NVTX markers in the execution context.
• ProfilingVerbosity getNvtxVerbosity () const noexcept
Get the NVTX verbosity of the execution context.
• void setAuxStreams (cudaStream t ∗auxStreams, int32 t nbStreams) noexcept
Set the auxiliary streams that TensorRT should launch kernels on in the next enqueueV3() call.
Protected Attributes
• apiv::VExecutionContext ∗ mImpl
Additional Inherited Members
9.59.1
Detailed Description
Context for executing inference using an engine, with functionally unsafe features.
Multiple execution contexts may exist for one ICudaEngine instance, allowing the same engine to be used for the exe-
cution of multiple batches simultaneously. If the engine supports dynamic shapes, each execution context in concurrent
use must use a separate optimization profile.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.59.2
Constructor & Destructor Documentation
9.59.2.1
∼IExecutionContext()
virtual nvinfer1::IExecutionContext::∼IExecutionContext ( ) [virtual], [default], [noexcept]
9.59.3
Member Function Documentation
298
Class Documentation
9.59.3.1
allInputDimensionsSpecified()
bool nvinfer1::IExecutionContext::allInputDimensionsSpecified ( ) const [inline], [noexcept]
Whether all dynamic dimensions of input tensors have been specified.
Returns
True if all dynamic dimensions of input tensors have been specified by calling setBindingDimensions().
Trivially true if network has no dynamically shaped input tensors.
Does not work with name-base interfaces eg. IExecutionContext::setInputShape(). Use IExecutionContext::inferShapes()
instead.
See also
setBindingDimensions(bindingIndex,dimensions)
9.59.3.2
allInputShapesSpecified()
bool nvinfer1::IExecutionContext::allInputShapesSpecified ( ) const [inline], [noexcept]
Whether all input shape bindings have been specified.
Returns
True if all input shape bindings have been specified by setInputShapeBinding().
Trivially true if network has no input shape bindings.
Does not work with name-base interfaces eg. IExecutionContext::setInputShape(). Use IExecutionContext::inferShapes()
instead.
See also
isShapeBinding(bindingIndex)
9.59.3.3
destroy()
TRT DEPRECATED void nvinfer1::IExecutionContext::destroy ( ) [inline], [noexcept]
Destroy this object.
Deprecated Deprecated in TRT 8.0. Superseded by delete.
9.59 nvinfer1::IExecutionContext Class Reference
299
Warning
Calling destroy on a managed pointer will result in a double-free error.
9.59.3.4
enqueue()
TRT DEPRECATED bool nvinfer1::IExecutionContext::enqueue (
int32 t batchSize,
void ∗const ∗ bindings,
cudaStream t stream,
cudaEvent t ∗ inputConsumed ) [inline], [noexcept]
Enqueue inference of a batch on a stream.
This method requires an array of input and output buffers. The mapping from tensor names to indices can be queried
using ICudaEngine::getBindingIndex()
Parameters
batchSize
The batch size. This is at most the max batch size value supplied to the builder when the engine
was built. If the network is created with NetworkDefinitionCreationFlag::kEXPLICIT BATCH
flag, please use enqueueV3() instead, and this batchSize argument has no effect.
bindings
An array of pointers to input and output buffers for the network.
stream
A cuda stream on which the inference kernels will be enqueued.
inputConsumed
An optional event which will be signaled when the input buffers can be refilled with new data.
Returns
True if the kernels were enqueued successfully.
Deprecated Deprecated in TensorRT
8.4.
Superseded by enqueueV2() if the network is created with
NetworkDefinitionCreationFlag::kEXPLICIT BATCH flag.
See also
ICudaEngine::getBindingIndex() ICudaEngine::getMaxBatchSize()
Warning
Calling enqueue() in from the same IExecutionContext object with different CUDA streams concurrently
results in undefined behavior. To perform inference concurrently in multiple streams, use one execution
context per stream.
This function will trigger layer resource updates if hasImplicitBatchDimension() returns true and batchSize
changes between subsequent calls, possibly resulting in performance bottlenecks.
300
Class Documentation
9.59.3.5
enqueueV2()
TRT DEPRECATED bool nvinfer1::IExecutionContext::enqueueV2 (
void ∗const ∗ bindings,
cudaStream t stream,
cudaEvent t ∗ inputConsumed ) [inline], [noexcept]
Enqueue inference on a stream.
This method requires an array of input and output buffers. The mapping from tensor names to indices can be queried
using ICudaEngine::getBindingIndex(). This method only works for execution contexts built with full dimension
networks.
Parameters
bindings
An array of pointers to input and output buffers for the network.
stream
A cuda stream on which the inference kernels will be enqueued
inputConsumed
An optional event which will be signaled when the input buffers can be refilled with new data
Returns
True if the kernels were enqueued successfully.
Deprecated Superseded by enqueueV3(). Deprecated in TensorRT 8.5
See also
ICudaEngine::getBindingIndex() ICudaEngine::getMaxBatchSize() IExecutionContext::enqueueV3()
Note
Calling enqueueV2() with a stream in CUDA graph capture mode has a known issue. If dynamic shapes are used,
the first enqueueV2() call after a setInputShapeBinding() call will cause failure in stream capture due to resource
allocation. Please call enqueueV2() once before capturing the graph.
Warning
Calling enqueueV2() in from the same IExecutionContext object with different CUDA streams concurrently
results in undefined behavior. To perform inference concurrently in multiple streams, use one execution
context per stream.
9.59.3.6
enqueueV3()
bool nvinfer1::IExecutionContext::enqueueV3 (
cudaStream t stream ) [inline], [noexcept]
Enqueue inference on a stream.
9.59 nvinfer1::IExecutionContext Class Reference
301
Parameters
stream
A cuda stream on which the inference kernels will be enqueued.
Returns
True if the kernels were enqueued successfully, false otherwise.
Modifying or releasing memory that has been registered for the tensors before stream synchronization or the event
passed to setInputConsumedEvent has been being triggered results in undefined behavior. Input tensor can be released
after the setInputConsumedEvent whereas output tensors require stream synchronization.
9.59.3.7
execute()
TRT DEPRECATED bool nvinfer1::IExecutionContext::execute (
int32 t batchSize,
void ∗const ∗ bindings ) [inline], [noexcept]
Synchronously execute inference on a batch.
This method requires an array of input and output buffers. The mapping from tensor names to indices can be queried
using ICudaEngine::getBindingIndex()
Parameters
batchSize
The batch size. This is at most the max batch size value supplied to the builder when the engine was
built. If the network is created with NetworkDefinitionCreationFlag::kEXPLICIT BATCH flag,
please use executeV2() instead, and this batchSize argument has no effect.
bindings
An array of pointers to input and output buffers for the network.
Returns
True if execution succeeded.
Deprecated Deprecated in TensorRT 8.4. Superseded by executeV2() if the network is created with NetworkDefinitionCreationFlag::kEX
flag.
Warning
This function will trigger layer resource updates if hasImplicitBatchDimension() returns true and batchSize
changes between subsequent calls, possibly resulting in performance bottlenecks.
See also
ICudaEngine::getBindingIndex() ICudaEngine::getMaxBatchSize()
302
Class Documentation
9.59.3.8
executeV2()
bool nvinfer1::IExecutionContext::executeV2 (
void ∗const ∗ bindings ) [inline], [noexcept]
Synchronously execute inference a network.
This method requires an array of input and output buffers. The mapping from tensor names to indices can be queried
using ICudaEngine::getBindingIndex(). This method only works for execution contexts built with full dimension
networks.
Parameters
bindings
An array of pointers to input and output buffers for the network.
Returns
True if execution succeeded.
See also
ICudaEngine::getBindingIndex() ICudaEngine::getMaxBatchSize()
9.59.3.9
getBindingDimensions()
TRT DEPRECATED Dims nvinfer1::IExecutionContext::getBindingDimensions (
int32 t bindingIndex ) const [inline], [noexcept]
Get the dynamic dimensions of a binding.
If the engine was built with an implicit batch dimension, same as ICudaEngine::getBindingDimensions.
If setBindingDimensions() has been called on this binding (or if there are no dynamic dimensions), all dimensions
will be positive. Otherwise, it is necessary to call setBindingDimensions() before enqueueV2() or executeV2() may be
called.
If the bindingIndex is out of range, an invalid Dims with nbDims == -1 is returned. The same invalid Dims will be
returned if the engine was not built with an implicit batch dimension and if the execution context is not currently
associated with a valid optimization profile (i.e. if getOptimizationProfile() returns -1).
If ICudaEngine::bindingIsInput(bindingIndex) is false, then both allInputDimensionsSpecified() and allInputShapesSpecified()
must be true before calling this method.
9.59 nvinfer1::IExecutionContext Class Reference
303
Returns
Currently selected binding dimensions
For backwards compatibility with earlier versions of TensorRT, a bindingIndex that does not belong to the current
profile is corrected as described for ICudaEngine::getProfileDimensions.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorShape().
See also
ICudaEngine::getProfileDimensions()
getTensorShape()
9.59.3.10 getDebugSync()
bool nvinfer1::IExecutionContext::getDebugSync ( ) const [inline], [noexcept]
Get the debug sync flag.
See also
setDebugSync()
9.59.3.11 getEngine()
ICudaEngine const & nvinfer1::IExecutionContext::getEngine ( ) const [inline], [noexcept]
Get the associated engine.
See also
ICudaEngine
304
Class Documentation
9.59.3.12 getEnqueueEmitsProfile()
bool nvinfer1::IExecutionContext::getEnqueueEmitsProfile ( ) const [inline], [noexcept]
Get the enqueueEmitsProfile state.
Returns
The enqueueEmitsProfile state.
See also
IExecutionContext::setEnqueueEmitsProfile()
9.59.3.13 getErrorRecorder()
IErrorRecorder ∗ nvinfer1::IExecutionContext::getErrorRecorder ( ) const [inline], [noexcept]
Get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A nullptr will be returned if an error handler has not
been set.
Returns
A pointer to the IErrorRecorder object that has been registered.
See also
setErrorRecorder()
9.59.3.14 getInputConsumedEvent()
cudaEvent t nvinfer1::IExecutionContext::getInputConsumedEvent ( ) const [inline], [noexcept]
The event associated with consuming the input.
Returns
The cuda event. Nullptr will be returned if the event is not set yet.
9.59.3.15 getMaxOutputSize()
int64 t nvinfer1::IExecutionContext::getMaxOutputSize (
char const ∗ tensorName ) const [inline], [noexcept]
Get upper bound on an output tensor's size, in bytes, based on the current optimization profile and input dimensions.
If the profile or input dimensions are not yet set, or the provided name does not map to an output, returns -1.
9.59 nvinfer1::IExecutionContext Class Reference
305
Parameters
tensorName
The name of an output tensor.
Returns
Upper bound in bytes.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.59.3.16 getName()
char const ∗ nvinfer1::IExecutionContext::getName ( ) const [inline], [noexcept]
Return the name of the execution context.
See also
setName()
9.59.3.17 getNvtxVerbosity()
ProfilingVerbosity nvinfer1::IExecutionContext::getNvtxVerbosity ( ) const [inline], [noexcept]
Get the NVTX verbosity of the execution context.
Returns
The current NVTX verbosity of the execution context.
See also
setNvtxVerbosity()
306
Class Documentation
9.59.3.18 getOptimizationProfile()
int32 t nvinfer1::IExecutionContext::getOptimizationProfile ( ) const [inline], [noexcept]
Get the index of the currently selected optimization profile.
If the profile index has not been set yet (implicitly to 0 if no other execution context has been set to profile 0, or
explicitly for all subsequent contexts), an invalid value of -1 will be returned and all calls to enqueueV2()/enqueue←↩
V3()/executeV2() will fail until a valid profile index has been set. This behavior is deprecated in TensorRT 8.6 and in
TensorRT 9.0, all profiles will default to optimization profile 0 and -1 will no longer be returned.
9.59.3.19 getOutputAllocator()
IOutputAllocator ∗ nvinfer1::IExecutionContext::getOutputAllocator (
char const ∗ tensorName ) const [inline], [noexcept]
Get output allocator associated with output tensor of given name, or nullptr if the provided name does not map to an
output tensor.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
See also
IOutputAllocator
9.59.3.20 getOutputTensorAddress()
void ∗ nvinfer1::IExecutionContext::getOutputTensorAddress (
char const ∗ tensorName ) const [inline], [noexcept]
Get memory address for given output.
Parameters
tensorName
The name of an output tensor.
Returns
Raw output data pointer (void∗) for given output tensor, or nullptr if the provided name does not map to an output
tensor.
If only a (void const∗) pointer is needed, an alternative is to call method getTensorAddress().
9.59 nvinfer1::IExecutionContext Class Reference
307
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
See also
getTensorAddress()
9.59.3.21 getPersistentCacheLimit()
size t nvinfer1::IExecutionContext::getPersistentCacheLimit ( ) const [inline], [noexcept]
Get the maximum size for persistent cache usage.
Returns
The size of the persistent cache limit
See also
setPersistentCacheLimit
9.59.3.22 getProfiler()
IProfiler ∗ nvinfer1::IExecutionContext::getProfiler ( ) const [inline], [noexcept]
Get the profiler.
See also
IProfiler setProfiler()
9.59.3.23 getShapeBinding()
TRT DEPRECATED bool nvinfer1::IExecutionContext::getShapeBinding (
int32 t bindingIndex,
int32 t ∗ data ) const [inline], [noexcept]
Get values of an input tensor required for shape calculations or an output tensor produced by shape calculations.
308
Class Documentation
Parameters
bindingIndex
index of an input or output tensor for which ICudaEngine::isShapeBinding(bindingIndex) is true.
data
pointer to where values will be written. The number of values written is the product of the
dimensions returned by getBindingDimensions(bindingIndex).
If ICudaEngine::bindingIsInput(bindingIndex) is false, then both allInputDimensionsSpecified() and allInputShapesSpecified()
must be true before calling this method. The method will also fail if no valid optimization profile has been set for the
current execution context, i.e. if getOptimizationProfile() returns -1.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorAddress() or getOutputTensorAddress().
See also
isShapeBinding() getTensorAddress() getOutputTensorAddress()
9.59.3.24 getStrides()
TRT DEPRECATED Dims nvinfer1::IExecutionContext::getStrides (
int32 t bindingIndex ) const [inline], [noexcept]
Return the strides of the buffer for the given binding.
The strides are in units of elements, not components or bytes. For example, for TensorFormat::kHWC8, a stride of one
spans 8 scalars.
Note that strides can be different for different execution contexts with dynamic shapes.
If the bindingIndex is invalid or there are dynamic dimensions that have not been set yet, returns Dims with
Dims::nbDims = -1.
Parameters
bindingIndex
The binding index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorStrides().
See also
getTensorStrides()
9.59 nvinfer1::IExecutionContext Class Reference
309
9.59.3.25 getTemporaryStorageAllocator()
IGpuAllocator ∗ nvinfer1::IExecutionContext::getTemporaryStorageAllocator ( ) const [inline],
[noexcept]
Get allocator set by setTemporaryStorageAllocator.
Returns a nullptr if a nullptr was passed with setTemporaryStorageAllocator().
9.59.3.26 getTensorAddress()
void const ∗ nvinfer1::IExecutionContext::getTensorAddress (
char const ∗ tensorName ) const [inline], [noexcept]
Get memory address bound to given input or output tensor, or nullptr if the provided name does not map to an input or
output tensor.
Parameters
tensorName
The name of an input or output tensor.
Use method getOutputTensorAddress() if a non-const pointer for an output tensor is required.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
See also
getOutputTensorAddress()
9.59.3.27 getTensorShape()
Dims nvinfer1::IExecutionContext::getTensorShape (
char const ∗ tensorName ) const [inline], [noexcept]
Return the shape of the given input or output.
Parameters
tensorName
The name of an input or output tensor.
Return Dims{-1, {}} if the provided name does not map to an input or output tensor. Otherwise return the shape of the
310
Class Documentation
input or output tensor.
A dimension in an input tensor will have a -1 wildcard value if all the following are true:
• setInputShape() has not yet been called for this tensor
• The dimension is a runtime dimension that is not implicitly constrained to be a single value.
A dimension in an output tensor will have a -1 wildcard value if the dimension depends on values of execution tensors
OR if all the following are true:
• It is a runtime dimension.
• setInputShape() has NOT been called for some input tensor(s) with a runtime shape.
• setTensorAddress() has NOT been called for some input tensor(s) with isShapeInferenceIO() = true.
An output tensor may also have -1 wildcard dimensions if its shape depends on values of tensors supplied to
enqueueV3().
If the request is for the shape of an output tensor with runtime dimensions, all input tensors with isShapeInferenceIO()
= true should have their value already set, since these values might be needed to compute the output shape.
Examples of an input dimension that is implicitly constrained to a single value:
• The optimization profile specifies equal min and max values.
• The dimension is named and only one value meets the optimization profile requirements for dimensions with
that name.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.59.3.28 getTensorStrides()
Dims nvinfer1::IExecutionContext::getTensorStrides (
char const ∗ tensorName ) const [inline], [noexcept]
Return the strides of the buffer for the given tensor name.
The strides are in units of elements, not components or bytes. For example, for TensorFormat::kHWC8, a stride of one
spans 8 scalars.
Note that strides can be different for different execution contexts with dynamic shapes.
If the provided name does not map to an input or output tensor, or there are dynamic dimensions that have not been set
yet, return Dims{-1, {}}
9.59 nvinfer1::IExecutionContext Class Reference
311
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.59.3.29 inferShapes()
int32 t nvinfer1::IExecutionContext::inferShapes (
int32 t nbMaxNames,
char const ∗∗ tensorNames ) [inline], [noexcept]
Run shape calculations.
Parameters
nbMaxNames
Maximum number of names to write to tensorNames. When the return value is a positive value n
and tensorNames != nullptr, the names of min(n,nbMaxNames) insufficiently specified input
tensors are written to tensorNames.
tensorNames
Buffer in which to place names of insufficiently specified input tensors.
Returns
0 on success. Positive value n if n input tensors were not sufficiently specified. -1 for other errors.
An input tensor is insufficiently specified if either of the following is true:
• It has dynamic dimensions and its runtime dimensions have not yet been specified via IExecutionContext::setInputShape.
• isShapeInferenceIO(t)=true and the tensor's address has not yet been set.
If an output tensor has isShapeInferenceIO(t)=true and its address has been specified, then its value is written.
Returns -1 if tensorNames == nullptr and nbMaxNames != 0. Returns -1 if nbMaxNames < 0. Returns -1 if a tensor's
dimensions are invalid, e.g. a tensor ends up with a negative dimension.
9.59.3.30 reportToProfiler()
bool nvinfer1::IExecutionContext::reportToProfiler ( ) const [inline], [noexcept]
Calculate layer timing info for the current optimization profile in IExecutionContext and update the profiler after one
iteration of inference launch.
312
Class Documentation
If IExecutionContext::getEnqueueEmitsProfile() returns true, the enqueue function will calculate layer timing implic-
itly if a profiler is provided. This function returns true and does nothing.
If IExecutionContext::getEnqueueEmitsProfile() returns false, the enqueue function will record the CUDA event timers
if a profiler is provided. But it will not perform the layer timing calculation. IExecutionContext::reportToProfiler()
needs to be called explicitly to calculate layer timing for the previous inference launch.
In the CUDA graph launch scenario, it will record the same set of CUDA events as in regular enqueue functions if the
graph is captured from an IExecutionContext with profiler enabled. This function needs to be called after graph launch
to report the layer timing info to the profiler.
Warning
profiling CUDA graphs is only available from CUDA 11.1 onwards.
reportToProfiler uses the stream of the previous enqueue call, so the stream must be live otherwise behavior
is undefined.
Returns
true if the call succeeded, else false (e.g. profiler not provided, in CUDA graph capture mode, etc.)
See also
IExecutionContext::setEnqueueEmitsProfile()
IExecutionContext::getEnqueueEmitsProfile()
9.59.3.31 setAuxStreams()
void nvinfer1::IExecutionContext::setAuxStreams (
cudaStream t ∗ auxStreams,
int32 t nbStreams ) [inline], [noexcept]
Set the auxiliary streams that TensorRT should launch kernels on in the next enqueueV3() call.
If set, TensorRT will launch the kernels that are supposed to run on the auxiliary streams using the streams provided
by the user with this API. If this API is not called before the enqueueV3() call, then TensorRT will use the auxiliary
streams created by TensorRT internally.
TensorRT will always insert event synchronizations between the main stream provided via enqueueV3() call and the
auxiliary streams:
• At the beginning of the enqueueV3() call, TensorRT will make sure that all the auxiliary streams wait on the
activities on the main stream.
• At the end of the enqueueV3() call, TensorRT will make sure that the main stream wait on the activities on all
the auxiliary streams.
9.59 nvinfer1::IExecutionContext Class Reference
313
Parameters
auxStreams
The pointer to an array of cudaStream t with the array length equal to nbStreams.
nbStreams
The number of auxiliary streams provided. If nbStreams is greater than
engine->getNbAuxStreams(), then only the first engine->getNbAuxStreams()
streams will be used. If nbStreams is less than engine->getNbAuxStreams(), such as
setting nbStreams to 0, then TensorRT will use the provided streams for the first nbStreams
auxiliary streams, and will create additional streams internally for the rest of the auxiliary streams.
Note
The provided auxiliary streams must not be the default stream and must all be different to avoid deadlocks.
See also
enqueueV3(), IBuilderConfig::setMaxAuxStreams(), ICudaEngine::getNbAuxStreams()
9.59.3.32 setBindingDimensions()
TRT DEPRECATED bool nvinfer1::IExecutionContext::setBindingDimensions (
int32 t bindingIndex,
Dims dimensions ) [inline], [noexcept]
Set the dynamic dimensions of an input binding.
Parameters
bindingIndex
index of an input tensor whose dimensions must be compatible with the network definition (i.e.
only the wildcard dimension -1 can be replaced with a new dimension >= 0).
dimensions
specifies the dimensions of the input tensor. It must be in the valid range for the currently selected
optimization profile, and the corresponding engine must not be safety-certified.
This method requires the engine to be built without an implicit batch dimension. This method will fail unless a valid
optimization profile is defined for the current execution context (getOptimizationProfile() must not be -1).
For all dynamic non-output bindings (which have at least one wildcard dimension of -1), this method needs
to be called before either enqueueV2() or executeV2() may be called. This can be checked using the method
allInputDimensionsSpecified().
Warning
This function will trigger layer resource updates on the next call of enqueueV2()/executeV2(), possibly
resulting in performance bottlenecks, if the dimensions are different than the previous set dimensions.
314
Class Documentation
Returns
false if an error occurs (e.g. bindingIndex is out of range for the currently selected optimization profile or
binding dimension is inconsistent with min-max range of the optimization profile), else true. Note that the
network can still be invalid for certain combinations of input shapes that lead to invalid output shapes. To
confirm the correctness of the network input shapes, check whether the output binding has valid dimensions
using getBindingDimensions() on the output bindingIndex.
Deprecated Deprecated in TensorRT 8.5. Superseded by setInputShape().
See also
setInputShape()
9.59.3.33 setDebugSync()
void nvinfer1::IExecutionContext::setDebugSync (
bool sync ) [inline], [noexcept]
Set the debug sync flag.
If this flag is set to true, the engine will log the successful execution for each kernel during executeV2(). It has no
effect when using enqueueV2()/enqueueV3().
See also
getDebugSync()
9.59.3.34 setDeviceMemory()
void nvinfer1::IExecutionContext::setDeviceMemory (
void ∗ memory ) [inline], [noexcept]
Set the device memory for use by this execution context.
The memory must be aligned with cuda memory alignment property (using cudaGetDeviceProperties()), and its size
must be at least that returned by getDeviceMemorySize(). Setting memory to nullptr is acceptable if getDevice←↩
MemorySize() returns 0. If using enqueueV2()/enqueueV3() to run the network, the memory is in use from the
invocation of enqueueV2()/enqueueV3() until network execution is complete. If using executeV2(), it is in use un-
til executeV2() returns. Releasing or otherwise using the memory for other purposes during this time will result in
undefined behavior.
See also
ICudaEngine::getDeviceMemorySize() ICudaEngine::createExecutionContextWithoutDeviceMemory()
9.59 nvinfer1::IExecutionContext Class Reference
315
9.59.3.35 setEnqueueEmitsProfile()
void nvinfer1::IExecutionContext::setEnqueueEmitsProfile (
bool enqueueEmitsProfile ) [inline], [noexcept]
Set whether enqueue emits layer timing to the profiler.
If set to true (default), enqueue is synchronous and does layer timing profiling implicitly if there is a profiler attached.
If set to false, enqueue will be asynchronous if there is a profiler attached. An extra method reportToProfiler() needs to
be called to obtain the profiling data and report to the profiler attached.
See also
IExecutionContext::getEnqueueEmitsProfile()
IExecutionContext::reportToProfiler()
9.59.3.36 setErrorRecorder()
void nvinfer1::IExecutionContext::setErrorRecorder (
IErrorRecorder ∗ recorder ) [inline], [noexcept]
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function
will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder
with the interface, resulting in a call to decRefCount if a recorder has been registered.
If an error recorder is not set, messages will be sent to the global log stream.
Parameters
recorder
The error recorder to register with this interface.
See also
getErrorRecorder()
9.59.3.37 setInputConsumedEvent()
bool nvinfer1::IExecutionContext::setInputConsumedEvent (
cudaEvent t event ) [inline], [noexcept]
Mark input as consumed.
316
Class Documentation
Parameters
event
The cuda event that is triggered after all input tensors have been consumed.
Warning
The set event must be valid during the inferece.
Returns
True on success, false if error occurred.
Passing event==nullptr removes whatever event was set, if any.
9.59.3.38 setInputShape()
bool nvinfer1::IExecutionContext::setInputShape (
char const ∗ tensorName,
Dims const & dims ) [inline], [noexcept]
Set shape of given input.
Parameters
tensorName
The name of an input tensor.
dims
The shape of an input tensor.
Returns
True on success, false if the provided name does not map to an input tensor, or if some other error occurred.
Each dimension must agree with the network dimension unless the latter was -1.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.59.3.39 setInputShapeBinding()
TRT DEPRECATED bool nvinfer1::IExecutionContext::setInputShapeBinding (
int32 t bindingIndex,
int32 t const ∗ data ) [inline], [noexcept]
Set values of input tensor required by shape calculations.
9.59 nvinfer1::IExecutionContext Class Reference
317
Parameters
bindingIndex
index of an input tensor for which ICudaEngine::isShapeBinding(bindingIndex) and
ICudaEngine::bindingIsInput(bindingIndex) are both true.
data
pointer to values of the input tensor. The number of values should be the product of the
dimensions returned by getBindingDimensions(bindingIndex).
If ICudaEngine::isShapeBinding(bindingIndex) and ICudaEngine::bindingIsInput(bindingIndex) are both true, this
method must be called before enqueueV2() or executeV2() may be called. This method will fail unless a valid op-
timization profile is defined for the current execution context (getOptimizationProfile() must not be -1).
Warning
This function will trigger layer resource updates on the next call of enqueueV2()/executeV2(), possibly
resulting in performance bottlenecks, if the shapes are different than the previous set shapes.
Returns
false if an error occurs (e.g. bindingIndex is out of range for the currently selected optimization profile or shape
data is inconsistent with min-max range of the optimization profile), else true. Note that the network can still be
invalid for certain combinations of input shapes that lead to invalid output shapes. To confirm the correctness of
the network input shapes, check whether the output binding has valid dimensions using getBindingDimensions()
on the output bindingIndex.
Deprecated Deprecated in TensorRT 8.5. Superseded by setInputTensorAddress() or setTensorAddress().
See also
setInputTensorAddress() setTensorAddress()
9.59.3.40 setInputTensorAddress()
bool nvinfer1::IExecutionContext::setInputTensorAddress (
char const ∗ tensorName,
void const ∗ data ) [inline], [noexcept]
Set memory address for given input.
Parameters
tensorName
The name of an input tensor.
data
The pointer (void const∗) to the const data owned by the user.
318
Class Documentation
Returns
True on success, false if the provided name does not map to an input tensor, does not meet alignment require-
ments, or some other error occurred.
Input addresses can also be set using method setTensorAddress, which requires a (void∗).
See description of method setTensorAddress() for alignment and data type constraints.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
See also
setTensorAddress()
9.59.3.41 setName()
void nvinfer1::IExecutionContext::setName (
char const ∗ name ) [inline], [noexcept]
Set the name of the execution context.
This method copies the name string.
Warning
The string name must be null-terminated, and be at most 4096 bytes including the terminator.
See also
getName()
9.59.3.42 setNvtxVerbosity()
bool nvinfer1::IExecutionContext::setNvtxVerbosity (
ProfilingVerbosity verbosity ) [inline], [noexcept]
Set the verbosity of the NVTX markers in the execution context.
Building with kDETAILED verbosity will generally increase latency in enqueueV2/enqueueV3(). Call this method to
select NVTX verbosity in this execution context at runtime.
The default is the verbosity with which the engine was built, and the verbosity may not be raised above that level.
This function does not affect how IEngineInspector interacts with the engine.
9.59 nvinfer1::IExecutionContext Class Reference
319
Parameters
verbosity
The verbosity of the NVTX markers.
Returns
True if the NVTX verbosity is set successfully. False if the provided verbosity level is higher than the profiling
verbosity of the corresponding engine.
See also
getNvtxVerbosity()
ICudaEngine::getProfilingVerbosity()
9.59.3.43 setOptimizationProfile()
TRT DEPRECATED bool nvinfer1::IExecutionContext::setOptimizationProfile (
int32 t profileIndex ) [inline], [noexcept]
Select an optimization profile for the current context.
Parameters
profileIndex
Index of the profile. It must lie between 0 and getEngine().getNbOptimizationProfiles() - 1
The selected profile will be used in subsequent calls to executeV2()/enqueueV2()/enqueueV3().
When an optimization profile is switched via this API, TensorRT may enqueue GPU memory copy operations required
to set up the new profile during the subsequent enqueueV2()/enqueueV3() operations. To avoid these calls during
enqueueV2()/enqueueV3(), use setOptimizationProfileAsync() instead.
If the associated CUDA engine does not have inputs with dynamic shapes, this method need not be called, in which
case the default profile index of 0 will be used (this is particularly the case for all safe engines).
setOptimizationProfile() must be called before calling setBindingDimensions() and setInputShapeBinding() for all
dynamic input tensors or input shape tensors, which in turn must be called before executeV2()/enqueueV2()/enqueue←↩
V3().
Warning
This function will trigger layer resource updates on the next call of enqueueV2()/enqueueV3()/executeV2(),
possibly resulting in performance bottlenecks.
320
Class Documentation
Returns
true if the call succeeded, else false (e.g. input out of range)
Deprecated Superseded by setOptimizationProfileAsync. Deprecated prior to TensorRT 8.0 and will be removed in
9.0.
See also
ICudaEngine::getNbOptimizationProfiles() IExecutionContext::setOptimizationProfileAsync()
9.59.3.44 setOptimizationProfileAsync()
bool nvinfer1::IExecutionContext::setOptimizationProfileAsync (
int32 t profileIndex,
cudaStream t stream ) [inline], [noexcept]
Select an optimization profile for the current context with async semantics.
Parameters
profileIndex
Index of the profile. The value must lie between 0 and getEngine().getNbOptimizationProfiles() - 1
stream
A cuda stream on which the cudaMemcpyAsyncs may be enqueued
When an optimization profile is switched via this API, TensorRT may require that data is copied via cudaMemcpy←↩
Async. It is the application’s responsibility to guarantee that synchronization between the profile sync stream and the
enqueue stream occurs.
The selected profile will be used in subsequent calls to executeV2()/enqueueV2()/enqueueV3(). If the associated
CUDA engine has inputs with dynamic shapes, the optimization profile must be set with its corresponding profileIndex
before calling execute or enqueue. If no execution context is assigned optimization profile 0 and a new context is
created for an engine, setOptimizationProfile(0) is called implicitly. This functionality is deprecated in TensorRT 8.6
and will instead default all optimization profiles to 0 starting in TensorRT 9.0.
If the associated CUDA engine does not have inputs with dynamic shapes, this method need not be called, in which
case the default profile index of 0 will be used.
setOptimizationProfileAsync() must be called before calling setBindingDimensions() and setInputShapeBinding()
for all dynamic input tensors or input shape tensors, which in turn must be called before executeV2()/enqueue←↩
V2()/enqueueV3().
9.59 nvinfer1::IExecutionContext Class Reference
321
Warning
This function will trigger layer resource updates on the next call of enqueueV2()/executeV2()/enqueueV3(),
possibly resulting in performance bottlenecks.
Not synchronizing the stream used at enqueue with the stream used to set optimization profile asyn-
chronously using this API will result in undefined behavior.
Returns
true if the call succeeded, else false (e.g. input out of range)
See also
ICudaEngine::getNbOptimizationProfiles()
IExecutionContext::setOptimizationProfile()
9.59.3.45 setOutputAllocator()
bool nvinfer1::IExecutionContext::setOutputAllocator (
char const ∗ tensorName,
IOutputAllocator ∗ outputAllocator ) [inline], [noexcept]
Set output allocator to use for output tensor of given name. Pass nullptr to outputAllocator to unset. The allocator is
called by enqueueV3().
Parameters
tensorName
The name of an output tensor.
outputAllocator
IOutputAllocator for the tensors.
Returns
True if success, false if the provided name does not map to an output or, if some other error occurred.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
See also
enqueueV3() IOutputAllocator
322
Class Documentation
9.59.3.46 setPersistentCacheLimit()
void nvinfer1::IExecutionContext::setPersistentCacheLimit (
size t size ) [inline], [noexcept]
Set the maximum size for persistent cache usage.
This function sets the maximum persistent L2 cache that this execution context may use for activation caching. Ac-
tivation caching is not supported on all architectures - see ”How TensorRT uses Memory” in the developer guide for
details
Parameters
size
the size of persistent cache limitation in bytes. The default is 0 Bytes.
See also
getPersistentCacheLimit
9.59.3.47 setProfiler()
void nvinfer1::IExecutionContext::setProfiler (
IProfiler ∗ profiler ) [inline], [noexcept]
Set the profiler.
See also
IProfiler getProfiler()
9.59.3.48 setTemporaryStorageAllocator()
bool nvinfer1::IExecutionContext::setTemporaryStorageAllocator (
IGpuAllocator ∗ allocator ) [inline], [noexcept]
Specify allocator to use for internal temporary storage.
This allocator is used only by enqueueV3() for temporary storage whose size cannot be predicted ahead of
enqueueV3(). It is not used for output tensors, because memory allocation for those is allocated by the allocator
set by setOutputAllocator(). All memory allocated is freed by the time enqueueV3() returns.
9.59 nvinfer1::IExecutionContext Class Reference
323
Parameters
allocator
pointer to allocator to use. Pass nullptr to revert to using TensorRT's default allocator.
Returns
True on success, false if error occurred.
See also
enqueueV3() setOutputAllocator()
9.59.3.49 setTensorAddress()
bool nvinfer1::IExecutionContext::setTensorAddress (
char const ∗ tensorName,
void ∗ data ) [inline], [noexcept]
Set memory address for given input or output tensor.
Parameters
tensorName
The name of an input or output tensor.
data
The pointer (void∗) to the data owned by the user.
Returns
True on success, false if error occurred.
An address defaults to nullptr. Pass data=nullptr to reset to the default state.
Return false if the provided name does not map to an input or output tensor.
If an input pointer has type (void const∗), use setInputTensorAddress() instead.
Before calling enqueueV3(), each input must have a non-null address and each output must have a non-null address or
an IOutputAllocator to set it later.
If the TensorLocation of the tensor is kHOST, the pointer must point to a host buffer of sufficient size. For shape
tensors, the only supported data type is int32 t. If the TensorLocation of the tensor is kDEVICE, the pointer must point
to a device buffer of sufficient size and alignment, or be nullptr if the tensor is an output tensor that will be allocated
by IOutputAllocator.
If getTensorShape(name) reports a -1 for any dimension of an output after all input shapes have been set, then to find
out the dimensions, use setOutputAllocator() to associate an IOutputAllocator to which the dimensions will be reported
when known.
Calling both setTensorAddress and setOutputAllocator() for the same output is allowed, and can be useful for preallo-
cating memory, and then reallocating if it's not big enough.
The pointer must have at least 256-byte alignment.
324
Class Documentation
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
See also
setInputTensorAddress() getTensorShape() setOutputAllocator() IOutputAllocator
9.59.4
Member Data Documentation
9.59.4.1
mImpl
apiv::VExecutionContext∗ nvinfer1::IExecutionContext::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInferRuntime.h
9.60
nvinfer1::safe::IExecutionContext Class Reference
Functionally safe context for executing inference using an engine.
#include <NvInferSafeRuntime.h>
Public Member Functions
• virtual ICudaEngine const & getEngine () const noexcept=0
Get the associated engine.
• virtual void setName (AsciiChar const ∗const name) noexcept=0
Set the name of the execution context.
• virtual AsciiChar const ∗ getName () const noexcept=0
Return the name of the execution context.
• virtual void setDeviceMemory (void ∗const memory) noexcept=0
Set the device memory for use by this execution context.
• virtual TRT DEPRECATED Dims getStrides (std::int32 t const bindingIndex) const noexcept=0
Return the strides of the buffer for the given binding.
• virtual void setErrorRecorder (IErrorRecorder ∗const recorder) noexcept=0
Set the ErrorRecorder for this interface.
• virtual IErrorRecorder ∗ getErrorRecorder () const noexcept=0
get the ErrorRecorder assigned to this interface.
9.60 nvinfer1::safe::IExecutionContext Class Reference
325
•
virtual TRT DEPRECATED bool enqueueV2 (void ∗const ∗const bindings, cudaStream t const stream, cuda←↩
Event t const ∗const inputConsumed) noexcept=0
Enqueue inference of a batch on a stream.
•
IExecutionContext ()=default
•
virtual ∼IExecutionContext () noexcept=default
•
IExecutionContext (IExecutionContext const &)=delete
•
IExecutionContext (IExecutionContext &&)=delete
•
IExecutionContext & operator= (IExecutionContext const &) &=delete
•
IExecutionContext & operator= (IExecutionContext &&) &=delete
•
virtual void setErrorBuffer (RuntimeErrorInformation ∗const buffer) noexcept=0
Set error buffer output for floating point errors.
•
virtual RuntimeErrorInformation ∗ getErrorBuffer () const noexcept=0
Get error buffer output for floating point errors.
•
virtual Dims getTensorStrides (AsciiChar const ∗tensorName) const noexcept=0
Return the strides of the buffer for the given tensor name.
•
virtual bool setInputTensorAddress (AsciiChar const ∗tensorName, void const ∗data) noexcept=0
Set memory address for given input tensor.
•
virtual bool setOutputTensorAddress (AsciiChar const ∗tensorName, void ∗data) noexcept=0
Set memory address for given output tensor.
•
virtual bool setInputConsumedEvent (cudaEvent t event) noexcept=0
Mark input as consumed.
•
virtual cudaEvent t getInputConsumedEvent () const noexcept=0
Return the event associated with consuming the input.
•
virtual void const ∗ getInputTensorAddress (AsciiChar const ∗tensorName) const noexcept=0
Get memory address for given input tensor.
•
virtual void ∗ getOutputTensorAddress (AsciiChar const ∗tensorName) const noexcept=0
Get memory address for given output tensor.
•
virtual bool enqueueV3 (cudaStream t stream) noexcept=0
Enqueue inference on a stream.
9.60.1
Detailed Description
Functionally safe context for executing inference using an engine.
Multiple safe execution contexts may exist for one safe::ICudaEngine instance, allowing the same engine to be used
for the execution of multiple inputs simultaneously.
Warning
Do not call the APIs of the same IExecutionContext from multiple threads at any given time. Each concur-
rent execution must have its own instance of an IExecutionContext.
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.60.2
Constructor & Destructor Documentation
326
Class Documentation
9.60.2.1
IExecutionContext() [1/3]
nvinfer1::safe::IExecutionContext::IExecutionContext ( ) [default]
9.60.2.2
∼IExecutionContext()
virtual nvinfer1::safe::IExecutionContext::∼IExecutionContext ( ) [virtual], [default], [noexcept]
9.60.2.3
IExecutionContext() [2/3]
nvinfer1::safe::IExecutionContext::IExecutionContext (
IExecutionContext const &
) [delete]
9.60.2.4
IExecutionContext() [3/3]
nvinfer1::safe::IExecutionContext::IExecutionContext (
IExecutionContext &&
) [delete]
9.60.3
Member Function Documentation
9.60.3.1
enqueueV2()
virtual TRT DEPRECATED bool nvinfer1::safe::IExecutionContext::enqueueV2 (
void ∗const ∗const bindings,
cudaStream t const stream,
cudaEvent t const ∗const inputConsumed ) [pure virtual], [noexcept]
Enqueue inference of a batch on a stream.
This method requires an array of input and output buffers. The mapping from tensor names to indices can be queried
using safe::ICudaEngine::getBindingIndex(). This method only works for an execution context built from a network
without an implicit batch dimension.
Parameters
bindings
An array of pointers to input and output buffers for the network.
stream
A cuda stream on which the inference kernels will be enqueued.
inputConsumed
nsorRT 8.6.11 API Reference
9.60 nvinfer1::safe::IExecutionContext Class Reference
327
Returns
True if the kernels were enqueued successfully.
Deprecated Deprecated in TensorRT 8.5. Superseded by enqueueV3().
See also
safe::IExecutionContext::enqueueV3()
Usage considerations
• Allowed context for the API call
- Thread-safe: No
9.60.3.2
enqueueV3()
virtual bool nvinfer1::safe::IExecutionContext::enqueueV3 (
cudaStream t stream ) [pure virtual], [noexcept]
Enqueue inference on a stream.
Modifying or releasing memory that has been registered for the tensors before stream synchronization or the event
passed to setInputConsumedEvent has been being triggered results in undefined behavior.
Parameters
stream
A cuda stream on which the inference kernels will be enqueued.
Returns
True on success, false if any execution error occurred.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
328
Class Documentation
9.60.3.3
getEngine()
virtual ICudaEngine const & nvinfer1::safe::IExecutionContext::getEngine ( ) const [pure virtual],
[noexcept]
Get the associated engine.
See also
safe::ICudaEngine
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.60.3.4
getErrorBuffer()
virtual RuntimeErrorInformation ∗ nvinfer1::safe::IExecutionContext::getErrorBuffer ( ) const
[pure virtual], [noexcept]
Get error buffer output for floating point errors.
Returns
Pointer to device memory to use as floating point error buffer or nullptr if not set.
See also
setErrorBuffer()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.60 nvinfer1::safe::IExecutionContext Class Reference
329
9.60.3.5
getErrorRecorder()
virtual IErrorRecorder ∗ nvinfer1::safe::IExecutionContext::getErrorRecorder ( ) const [pure
virtual], [noexcept]
get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A default error recorder does not exist, so a nullptr will
be returned if setErrorRecorder has not been called.
Returns
A pointer to the IErrorRecorder object that has been registered.
See also
setErrorRecorder()
Usage considerations
• Allowed context for the API call
- Thread-safe: No
9.60.3.6
getInputConsumedEvent()
virtual cudaEvent t nvinfer1::safe::IExecutionContext::getInputConsumedEvent ( ) const [pure
virtual], [noexcept]
Return the event associated with consuming the input.
Returns
The cuda event, nullptr will be returned if the event is not set yet.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.60.3.7
getInputTensorAddress()
virtual void const ∗ nvinfer1::safe::IExecutionContext::getInputTensorAddress (
AsciiChar const ∗ tensorName ) const [pure virtual], [noexcept]
Get memory address for given input tensor.
330
Class Documentation
Parameters
tensorName
The name of an input tensor.
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
Returns
The memory address for the given input tensor. nullptr will be returned if (1) name is not the name of an input
tensor, or (2) name is nullptr, or (3) name exceeds the string length limit, or (4) the memory address for the given
input tensor is not set yet.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.60.3.8
getName()
virtual AsciiChar const ∗ nvinfer1::safe::IExecutionContext::getName ( ) const [pure virtual],
[noexcept]
Return the name of the execution context.
See also
setName()
Usage considerations
• Allowed context for the API call
- Thread-safe: No
9.60.3.9
getOutputTensorAddress()
virtual void ∗ nvinfer1::safe::IExecutionContext::getOutputTensorAddress (
AsciiChar const ∗ tensorName ) const [pure virtual], [noexcept]
Get memory address for given output tensor.
9.60 nvinfer1::safe::IExecutionContext Class Reference
331
Parameters
tensorName
The name of an output tensor.
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
Returns
Raw output data pointer (void∗) for given output tensor, return nullptr if (1) name is not the name of an output
tensor, or (2) name is nullptr, or (3) name exceeds the string length limit, or (4) the memory address for the given
output tensor is not set yet.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.60.3.10 getStrides()
virtual TRT DEPRECATED Dims nvinfer1::safe::IExecutionContext::getStrides (
std::int32 t const bindingIndex ) const [pure virtual], [noexcept]
Return the strides of the buffer for the given binding.
Parameters
bindingIndex
The binding index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorStrides().
See also
safe::IExecutionContext::getTensorStrides()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
|
||
|
|
|