|
|
182
Class Documentation
See also
getMaxAuxStreams(), ICudaEngine::getNbAuxStreams(), IExecutionContext::setAuxStreams()
9.42.3.49 setMaxWorkspaceSize()
TRT DEPRECATED void nvinfer1::IBuilderConfig::setMaxWorkspaceSize (
std::size t workspaceSize ) [inline], [noexcept]
Set the maximum workspace size.
Parameters
workspaceSize
The maximum GPU temporary memory which the engine can use at execution time.
See also
getMaxWorkspaceSize()
Deprecated Deprecated in TensorRT 8.3. Superseded by IBuilderConfig::setMemoryPoolLimit() with MemoryPoolType::kWORKSPAC
9.42.3.50 setMemoryPoolLimit()
void nvinfer1::IBuilderConfig::setMemoryPoolLimit (
MemoryPoolType pool,
std::size t poolSize ) [inline], [noexcept]
Set the memory size for the memory pool.
TensorRT layers access different memory pools depending on the operation. This function sets in the IBuilderConfig
the size limit, specified by poolSize, for the corresponding memory pool, specified by pool. TensorRT will build
a plan file that is constrained by these limits or report which constraint caused the failure.
If the size of the pool, specified by poolSize, fails to meet the size requirements for the pool, this function does
nothing and emits the recoverable error, ErrorCode::kINVALID ARGUMENT, to the registered IErrorRecorder.
If the size of the pool is larger than the maximum possible value for the configuration, this function does nothing and
emits ErrorCode::kUNSUPPORTED STATE.
If the pool does not exist on the requested device type when building the network, a warning is emitted to the logger,
and the memory pool value is ignored.
Refer to MemoryPoolType to see the size requirements for each pool.
9.42 nvinfer1::IBuilderConfig Class Reference
183
Parameters
pool
The memory pool to limit the available memory for.
poolSize
The size of the pool in bytes.
See also
getMemoryPoolLimit, MemoryPoolType
9.42.3.51 setMinTimingIterations()
virtual TRT DEPRECATED void nvinfer1::IBuilderConfig::setMinTimingIterations (
int32 t minTiming ) [inline], [virtual], [noexcept]
Set the number of minimization iterations used when timing layers.
When timing layers, the builder minimizes over a set of average times for layer execution. This parameter controls
the number of iterations used in minimization. The builder may sometimes run layers for more iterations to improve
timing accuracy if this parameter is set to a small value and the runtime of the layer is short.
See also
getMinTimingIterations()
Deprecated Deprecated in TensorRT 8.4. Superseded by setAvgTimingIterations().
9.42.3.52 setPluginsToSerialize()
void nvinfer1::IBuilderConfig::setPluginsToSerialize (
char const ∗const ∗ paths,
int32 t nbPaths ) [inline], [noexcept]
Set the plugin libraries to be serialized with version-compatible engines.
Each entry in the list of libraries must be unique.
Parameters
paths
The paths of plugin libraries.
nbPaths
The number of paths.
184
Class Documentation
9.42.3.53 setPreviewFeature()
void nvinfer1::IBuilderConfig::setPreviewFeature (
PreviewFeature feature,
bool enable ) [inline], [noexcept]
Enable or disable a specific preview feature.
Allows enabling or disabling experimental features, which are not enabled by default in the current release.
Refer to PreviewFeature for additional information, and a list of the available features.
Parameters
feature
the feature to enable / disable
enable
true for enable, false for disable
See also
PreviewFeature, getPreviewFeature
9.42.3.54 setProfileStream()
void nvinfer1::IBuilderConfig::setProfileStream (
const cudaStream t stream ) [inline], [noexcept]
Set the cuda stream that is used to profile this network.
Parameters
stream
The cuda stream used for profiling by the builder.
See also
getProfileStream()
9.42.3.55 setProfilingVerbosity()
void nvinfer1::IBuilderConfig::setProfilingVerbosity (
ProfilingVerbosity verbosity ) [inline], [noexcept]
9.42 nvinfer1::IBuilderConfig Class Reference
185
Set verbosity level of layer information exposed in NVTX annotations and IEngineInspector.
Control how much layer information will be exposed in NVTX annotations and IEngineInspector.
See also
ProfilingVerbosity, getProfilingVerbosity(), IEngineInspector
9.42.3.56 setQuantizationFlag()
void nvinfer1::IBuilderConfig::setQuantizationFlag (
QuantizationFlag flag ) [inline], [noexcept]
Set a single quantization flag.
Add the input quantization flag to the already enabled quantization flags.
See also
setQuantizationFlags()
9.42.3.57 setQuantizationFlags()
void nvinfer1::IBuilderConfig::setQuantizationFlags (
QuantizationFlags flags ) [inline], [noexcept]
Set the quantization flags.
The flags are listed in the QuantizationFlag enum. The flags set configuration options to quantize the network in int8.
Parameters
flags
The quantization flags.
Note
This function will override the previous set flags, rather than bitwise ORing the new flag.
See also
getQuantizationFlags()
186
Class Documentation
9.42.3.58 setTacticSources()
bool nvinfer1::IBuilderConfig::setTacticSources (
TacticSources tacticSources ) [inline], [noexcept]
Set tactic sources.
This bitset controls which tactic sources TensorRT is allowed to use for tactic selection.
Multiple tactic sources may be combined with a bitwise OR operation. For example, to enable cublas and cublasLt as
tactic sources, use a value of:
1U << static cast<uint32 t>(TacticSource::kCUBLAS) | 1U << static cast<uint32 t>(TacticSource::kCUBLAS←↩
LT)
See also
getTacticSources
Returns
true if the tactic sources in the build configuration were updated. The tactic sources in the build configuration
will not be updated if the provided value is invalid.
9.42.3.59 setTimingCache()
bool nvinfer1::IBuilderConfig::setTimingCache (
ITimingCache const & cache,
bool ignoreMismatch ) [inline], [noexcept]
Attach a timing cache to IBuilderConfig.
The timing cache has verification header to make sure the provided cache can be used in current environment. A
failure will be reported if the CUDA device property in the provided cache is different from current environment.
ignoreMismatch = true skips strict verification and allows loading cache created from a different device.
The cache must not be destroyed until after the engine is built.
Parameters
cache
the timing cache to be used
ignoreMismatch
whether or not allow using a cache that contains different CUDA device property
9.43 nvcaffeparser1::ICaffeParser Class Reference
187
Returns
true if set successfully, false otherwise
Warning
Using cache generated from devices with different CUDA device properties may lead to func-
tional/performance bugs.
9.42.4
Member Data Documentation
9.42.4.1
mImpl
apiv::VBuilderConfig∗ nvinfer1::IBuilderConfig::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.43
nvcaffeparser1::ICaffeParser Class Reference
Class used for parsing Caffe models.
#include <NvCaffeParser.h>
Public Member Functions
• virtual IBlobNameToTensor const ∗ parse (char const ∗deploy, char const ∗model, nvinfer1::INetworkDefinition
&network, nvinfer1::DataType weightType) noexcept=0
Parse a prototxt file and a binaryproto Caffe model to extract network definition and weights associated with the network,
respectively.
• virtual IBlobNameToTensor const ∗ parseBuffers (uint8 t const ∗deployBuffer, std::size t deployLength, uint8←↩
t const ∗modelBuffer, std::size t modelLength, nvinfer1::INetworkDefinition &network, nvinfer1::DataType
weightType) noexcept=0
Parse a deploy prototxt and a binaryproto Caffe model from memory buffers to extract network definition and weights
associated with the network, respectively.
• virtual IBinaryProtoBlob ∗ parseBinaryProto (char const ∗fileName) noexcept=0
Parse and extract data stored in binaryproto file.
• virtual void setProtobufBufferSize (size t size) noexcept=0
Set buffer size for the parsing and storage of the learned model.
• virtual TRT DEPRECATED void destroy () noexcept=0
Destroy this ICaffeParser object.
• virtual void setPluginFactoryV2 (IPluginFactoryV2 ∗factory) noexcept=0
Set the IPluginFactoryV2 used to create the user defined pluginV2 objects.
• virtual void setPluginNamespace (char const ∗libNamespace) noexcept=0
Set the namespace used to lookup and create plugins in the network.
• virtual ∼ICaffeParser () noexcept=default
• virtual void setErrorRecorder (nvinfer1::IErrorRecorder ∗recorder) noexcept=0
Set the ErrorRecorder for this interface.
• virtual nvinfer1::IErrorRecorder ∗ getErrorRecorder () const noexcept=0
get the ErrorRecorder assigned to this interface.
188
Class Documentation
9.43.1
Detailed Description
Class used for parsing Caffe models.
Allows users to export models trained using Caffe to TRT.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.43.2
Constructor & Destructor Documentation
9.43.2.1
∼ICaffeParser()
virtual nvcaffeparser1::ICaffeParser::∼ICaffeParser ( ) [virtual], [default], [noexcept]
9.43.3
Member Function Documentation
9.43.3.1
destroy()
virtual TRT DEPRECATED void nvcaffeparser1::ICaffeParser::destroy ( ) [pure virtual], [noexcept]
Destroy this ICaffeParser object.
Deprecated Deprecated in TensorRT 8.0. Superseded by delete.
Warning
Calling destroy on a managed pointer will result in a double-free error.
9.43 nvcaffeparser1::ICaffeParser Class Reference
189
9.43.3.2
getErrorRecorder()
virtual nvinfer1::IErrorRecorder ∗ nvcaffeparser1::ICaffeParser::getErrorRecorder ( ) const [pure
virtual], [noexcept]
get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A nullptr will be returned if setErrorRecorder has not
been called.
Returns
A pointer to the IErrorRecorder object that has been registered.
See also
setErrorRecorder()
9.43.3.3
parse()
virtual IBlobNameToTensor const ∗ nvcaffeparser1::ICaffeParser::parse (
char const ∗ deploy,
char const ∗ model,
nvinfer1::INetworkDefinition & network,
nvinfer1::DataType weightType ) [pure virtual], [noexcept]
Parse a prototxt file and a binaryproto Caffe model to extract network definition and weights associated with the
network, respectively.
Parameters
deploy
The plain text, prototxt file used to define the network definition.
model
The binaryproto Caffe model that contains the weights associated with the network.
network
Network in which the CaffeParser will fill the layers.
weightType
The type to which the weights will transformed.
Returns
A pointer to an IBlobNameToTensor object that contains the extracted data.
See also
nvcaffeparser1::IBlobNameToTensor
190
Class Documentation
9.43.3.4
parseBinaryProto()
virtual IBinaryProtoBlob ∗ nvcaffeparser1::ICaffeParser::parseBinaryProto (
char const ∗ fileName ) [pure virtual], [noexcept]
Parse and extract data stored in binaryproto file.
The binaryproto file contains data stored in a binary blob. parseBinaryProto() converts it to an IBinaryProtoBlob object
which gives the user access to the data and meta-data about data.
Parameters
fileName
Path to file containing binary proto.
Returns
A pointer to an IBinaryProtoBlob object that contains the extracted data.
See also
nvcaffeparser1::IBinaryProtoBlob
9.43.3.5
parseBuffers()
virtual IBlobNameToTensor const ∗ nvcaffeparser1::ICaffeParser::parseBuffers (
uint8 t const ∗ deployBuffer,
std::size t deployLength,
uint8 t const ∗ modelBuffer,
std::size t modelLength,
nvinfer1::INetworkDefinition & network,
nvinfer1::DataType weightType ) [pure virtual], [noexcept]
Parse a deploy prototxt and a binaryproto Caffe model from memory buffers to extract network definition and weights
associated with the network, respectively.
Parameters
deployBuffer
The plain text deploy prototxt used to define the network definition.
deployLength
The length of the deploy buffer.
modelBuffer
The binaryproto Caffe memory buffer that contains the weights associated with the network.
modelLength
The length of the model buffer.
network
Network in which the CaffeParser will fill the layers.
weightType
The type to which the weights will transformed.
9.43 nvcaffeparser1::ICaffeParser Class Reference
191
Returns
A pointer to an IBlobNameToTensor object that contains the extracted data.
See also
nvcaffeparser1::IBlobNameToTensor
9.43.3.6
setErrorRecorder()
virtual void nvcaffeparser1::ICaffeParser::setErrorRecorder (
nvinfer1::IErrorRecorder ∗ recorder ) [pure virtual], [noexcept]
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function
will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder
with the interface, resulting in a call to decRefCount if a recorder has been registered.
If an error recorder is not set, messages will be sent to the global log stream.
Parameters
recorder
The error recorder to register with this interface.
See also
getErrorRecorder()
9.43.3.7
setPluginFactoryV2()
virtual void nvcaffeparser1::ICaffeParser::setPluginFactoryV2 (
IPluginFactoryV2 ∗ factory ) [pure virtual], [noexcept]
Set the IPluginFactoryV2 used to create the user defined pluginV2 objects.
Parameters
factory
Pointer to an instance of the user implementation of IPluginFactoryV2.
192
Class Documentation
9.43.3.8
setPluginNamespace()
virtual void nvcaffeparser1::ICaffeParser::setPluginNamespace (
char const ∗ libNamespace ) [pure virtual], [noexcept]
Set the namespace used to lookup and create plugins in the network.
9.43.3.9
setProtobufBufferSize()
virtual void nvcaffeparser1::ICaffeParser::setProtobufBufferSize (
size t size ) [pure virtual], [noexcept]
Set buffer size for the parsing and storage of the learned model.
Parameters
size
The size of the buffer specified as the number of bytes.
Note
Default size is 2∧30 bytes.
The documentation for this class was generated from the following file:
• NvCaffeParser.h
9.44
nvinfer1::ICastLayer Class Reference
A cast layer in a network.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::ICastLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::ICastLayer
9.44 nvinfer1::ICastLayer Class Reference
193
Public Member Functions
• void setToType (DataType toType) noexcept
Set cast layer output type.
• DataType getToType () const noexcept
Return cast layer output type.
Protected Member Functions
• virtual ∼ICastLayer () noexcept=default
Protected Attributes
• apiv::VCastLayer ∗ mImpl
9.44.1
Detailed Description
A cast layer in a network.
This layer casts a given tensor to the datatype specified by toType.
9.44.2
Constructor & Destructor Documentation
9.44.2.1
∼ICastLayer()
virtual nvinfer1::ICastLayer::∼ICastLayer ( ) [protected], [virtual], [default], [noexcept]
9.44.3
Member Function Documentation
9.44.3.1
getToType()
DataType nvinfer1::ICastLayer::getToType ( ) const [inline], [noexcept]
Return cast layer output type.
194
Class Documentation
9.44.3.2
setToType()
void nvinfer1::ICastLayer::setToType (
DataType toType ) [inline], [noexcept]
Set cast layer output type.
9.44.4
Member Data Documentation
9.44.4.1
mImpl
apiv::VCastLayer∗ nvinfer1::ICastLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.45
nvinfer1::IConcatenationLayer Class Reference
A concatenation layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IConcatenationLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IConcatenationLayer
Public Member Functions
• void setAxis (int32 t axis) noexcept
Set the axis along which concatenation occurs.
• int32 t getAxis () const noexcept
Get the axis along which concatenation occurs.
9.45 nvinfer1::IConcatenationLayer Class Reference
195
Protected Member Functions
• virtual ∼IConcatenationLayer () noexcept=default
Protected Attributes
• apiv::VConcatenationLayer ∗ mImpl
9.45.1
Detailed Description
A concatenation layer in a network definition.
The output dimension along the concatenation axis is the sum of the corresponding input dimensions. Every other
output dimension is the same as the corresponding dimension of the inputs.
Warning
All tensors must have the same dimensions except along the concatenation axis.
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.45.2
Constructor & Destructor Documentation
9.45.2.1
∼IConcatenationLayer()
virtual nvinfer1::IConcatenationLayer::∼IConcatenationLayer ( ) [protected], [virtual], [default],
[noexcept]
9.45.3
Member Function Documentation
9.45.3.1
getAxis()
int32 t nvinfer1::IConcatenationLayer::getAxis ( ) const [inline], [noexcept]
Get the axis along which concatenation occurs.
See also
setAxis()
196
Class Documentation
9.45.3.2
setAxis()
void nvinfer1::IConcatenationLayer::setAxis (
int32 t axis ) [inline], [noexcept]
Set the axis along which concatenation occurs.
The default axis is the number of tensor dimensions minus three, or zero if the tensor has fewer than three dimensions.
For example, for a tensor with dimensions NCHW, it is C. For implicit batch mode, the number of tensor dimensions
does NOT include the implicit batch dimension.
When running this layer on the DLA, the concatenation axis must be the third to last axis, e.g. C if tensor dimensions
are NCHW.
Parameters
axis
The axis along which concatenation occurs.
9.45.4
Member Data Documentation
9.45.4.1
mImpl
apiv::VConcatenationLayer∗ nvinfer1::IConcatenationLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.46
nvinfer1::IConditionLayer Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IConditionLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IIfConditionalBoundaryLayer
nvinfer1::IConditionLayer
9.47 nvinfer1::consistency::IConsistencyChecker Class Reference
197
Protected Member Functions
• virtual ∼IConditionLayer () noexcept=default
Protected Attributes
• apiv::VConditionLayer ∗ mImpl
Additional Inherited Members
9.46.1
Detailed Description
This layer represents a condition input to an IIfConditional.
9.46.2
Constructor & Destructor Documentation
9.46.2.1
∼IConditionLayer()
virtual nvinfer1::IConditionLayer::∼IConditionLayer ( ) [protected], [virtual], [default], [noexcept]
9.46.3
Member Data Documentation
9.46.3.1
mImpl
apiv::VConditionLayer∗ nvinfer1::IConditionLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.47
nvinfer1::consistency::IConsistencyChecker Class Reference
Validates a serialized engine blob.
#include <NvInferConsistency.h>
198
Class Documentation
Public Member Functions
• bool validate () const noexcept
Check that a blob that was input to createConsistencyChecker method represents a valid engine.
• virtual ∼IConsistencyChecker ()=default
De-allocates any internally allocated memory.
Protected Member Functions
• IConsistencyChecker ()=default
• IConsistencyChecker (IConsistencyChecker const &other)=delete
• IConsistencyChecker & operator= (IConsistencyChecker const &other)=delete
• IConsistencyChecker (IConsistencyChecker &&other)=delete
• IConsistencyChecker & operator= (IConsistencyChecker &&other)=delete
Protected Attributes
• apiv::VConsistencyChecker ∗ mImpl
9.47.1
Detailed Description
Validates a serialized engine blob.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.47.2
Constructor & Destructor Documentation
9.47.2.1
∼IConsistencyChecker()
virtual nvinfer1::consistency::IConsistencyChecker::∼IConsistencyChecker ( ) [virtual], [default]
De-allocates any internally allocated memory.
9.47.2.2
IConsistencyChecker() [1/3]
nvinfer1::consistency::IConsistencyChecker::IConsistencyChecker ( ) [protected], [default]
9.47 nvinfer1::consistency::IConsistencyChecker Class Reference
199
9.47.2.3
IConsistencyChecker() [2/3]
nvinfer1::consistency::IConsistencyChecker::IConsistencyChecker (
IConsistencyChecker const & other ) [protected], [delete]
9.47.2.4
IConsistencyChecker() [3/3]
nvinfer1::consistency::IConsistencyChecker::IConsistencyChecker (
IConsistencyChecker && other ) [protected], [delete]
9.47.3
Member Function Documentation
9.47.3.1
operator=() [1/2]
IConsistencyChecker & nvinfer1::consistency::IConsistencyChecker::operator= (
IConsistencyChecker && other ) [protected], [delete]
9.47.3.2
operator=() [2/2]
IConsistencyChecker & nvinfer1::consistency::IConsistencyChecker::operator= (
IConsistencyChecker const & other ) [protected], [delete]
9.47.3.3
validate()
bool nvinfer1::consistency::IConsistencyChecker::validate ( ) const [inline], [noexcept]
Check that a blob that was input to createConsistencyChecker method represents a valid engine.
Returns
true if the original blob encoded an engine that belongs to valid engine domain with target capability
EngineCapability::kSAFETY, false otherwise.
9.47.4
Member Data Documentation
200
Class Documentation
9.47.4.1
mImpl
apiv::VConsistencyChecker∗ nvinfer1::consistency::IConsistencyChecker::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInferConsistency.h
9.48
nvinfer1::IConstantLayer Class Reference
Layer that represents a constant value.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IConstantLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IConstantLayer
Public Member Functions
• void setWeights (Weights weights) noexcept
Set the weights for the layer.
• Weights getWeights () const noexcept
Get the weights for the layer.
• void setDimensions (Dims dimensions) noexcept
Set the dimensions for the layer.
• Dims getDimensions () const noexcept
Get the dimensions for the layer.
Protected Member Functions
• virtual ∼IConstantLayer () noexcept=default
Protected Attributes
• apiv::VConstantLayer ∗ mImpl
9.48.1
Detailed Description
Layer that represents a constant value.
Note
This layer does not support boolean types.
9.48 nvinfer1::IConstantLayer Class Reference
201
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.48.2
Constructor & Destructor Documentation
9.48.2.1
∼IConstantLayer()
virtual nvinfer1::IConstantLayer::∼IConstantLayer ( ) [protected], [virtual], [default], [noexcept]
9.48.3
Member Function Documentation
9.48.3.1
getDimensions()
Dims nvinfer1::IConstantLayer::getDimensions ( ) const [inline], [noexcept]
Get the dimensions for the layer.
Returns
the dimensions for the layer
See also
getDimensions
9.48.3.2
getWeights()
Weights nvinfer1::IConstantLayer::getWeights ( ) const [inline], [noexcept]
Get the weights for the layer.
See also
setWeights
9.48.3.3
setDimensions()
void nvinfer1::IConstantLayer::setDimensions (
Dims dimensions ) [inline], [noexcept]
Set the dimensions for the layer.
202
Class Documentation
Parameters
dimensions
The dimensions of the layer
See also
setDimensions
9.48.3.4
setWeights()
void nvinfer1::IConstantLayer::setWeights (
Weights weights ) [inline], [noexcept]
Set the weights for the layer.
If weights.type is DataType::kINT32, the output is a tensor of 32-bit indices. Otherwise the output is a tensor of real
values and the output type will be follow TensorRT's normal precision rules.
See also
getWeights()
9.48.4
Member Data Documentation
9.48.4.1
mImpl
apiv::VConstantLayer∗ nvinfer1::IConstantLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.49
nvinfer1::IConvolutionLayer Class Reference
A convolution layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IConvolutionLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IConvolutionLayer
9.49 nvinfer1::IConvolutionLayer Class Reference
203
Public Member Functions
•
TRT DEPRECATED void setKernelSize (DimsHW kernelSize) noexcept
Set the HW kernel size of the convolution.
•
TRT DEPRECATED DimsHW getKernelSize () const noexcept
Get the HW kernel size of the convolution.
•
void setNbOutputMaps (int32 t nbOutputMaps) noexcept
Set the number of output maps for the convolution.
•
int32 t getNbOutputMaps () const noexcept
Get the number of output maps for the convolution.
•
TRT DEPRECATED void setStride (DimsHW stride) noexcept
Get the stride of the convolution.
•
TRT DEPRECATED DimsHW getStride () const noexcept
Get the stride of the convolution.
•
TRT DEPRECATED void setPadding (DimsHW padding) noexcept
Set the padding of the convolution.
•
TRT DEPRECATED DimsHW getPadding () const noexcept
Get the padding of the convolution. If the padding is asymmetric, the pre-padding is returned.
•
void setNbGroups (int32 t nbGroups) noexcept
Set the number of groups for a convolution.
•
int32 t getNbGroups () const noexcept
Get the number of groups of the convolution.
•
void setKernelWeights (Weights weights) noexcept
Set the kernel weights for the convolution.
•
Weights getKernelWeights () const noexcept
Get the kernel weights of the convolution.
•
void setBiasWeights (Weights weights) noexcept
Set the bias weights for the convolution.
•
Weights getBiasWeights () const noexcept
Get the bias weights for the convolution.
•
TRT DEPRECATED void setDilation (DimsHW dilation) noexcept
Set the dilation for a convolution.
•
TRT DEPRECATED DimsHW getDilation () const noexcept
Get the dilation for a convolution.
•
void setPrePadding (Dims padding) noexcept
Set the multi-dimension pre-padding of the convolution.
•
Dims getPrePadding () const noexcept
Get the pre-padding.
•
void setPostPadding (Dims padding) noexcept
Set the multi-dimension post-padding of the convolution.
•
Dims getPostPadding () const noexcept
Get the post-padding.
•
void setPaddingMode (PaddingMode paddingMode) noexcept
Set the padding mode.
•
PaddingMode getPaddingMode () const noexcept
Get the padding mode.
•
void setKernelSizeNd (Dims kernelSize) noexcept
204
Class Documentation
Set the multi-dimension kernel size of the convolution.
• Dims getKernelSizeNd () const noexcept
Get the multi-dimension kernel size of the convolution.
• void setStrideNd (Dims stride) noexcept
Set the multi-dimension stride of the convolution.
• Dims getStrideNd () const noexcept
Get the multi-dimension stride of the convolution.
• void setPaddingNd (Dims padding) noexcept
Set the multi-dimension padding of the convolution.
• Dims getPaddingNd () const noexcept
Get the multi-dimension padding of the convolution.
• void setDilationNd (Dims dilation) noexcept
Set the multi-dimension dilation of the convolution.
• Dims getDilationNd () const noexcept
Get the multi-dimension dilation of the convolution.
• void setInput (int32 t index, ITensor &tensor) noexcept
Append or replace an input of this layer with a specific tensor.
Protected Member Functions
• virtual ∼IConvolutionLayer () noexcept=default
Protected Attributes
• apiv::VConvolutionLayer ∗ mImpl
9.49.1
Detailed Description
A convolution layer in a network definition.
This layer performs a correlation operation between 3-dimensional filter with a 4-dimensional tensor to produce another
4-dimensional tensor.
An optional bias argument is supported, which adds a per-channel constant to each value in the output.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.49.2
Constructor & Destructor Documentation
9.49 nvinfer1::IConvolutionLayer Class Reference
205
9.49.2.1
∼IConvolutionLayer()
virtual nvinfer1::IConvolutionLayer::∼IConvolutionLayer ( ) [protected], [virtual], [default],
[noexcept]
9.49.3
Member Function Documentation
9.49.3.1
getBiasWeights()
Weights nvinfer1::IConvolutionLayer::getBiasWeights ( ) const [inline], [noexcept]
Get the bias weights for the convolution.
See also
setBiasWeights()
9.49.3.2
getDilation()
TRT DEPRECATED DimsHW nvinfer1::IConvolutionLayer::getDilation ( ) const [inline], [noexcept]
Get the dilation for a convolution.
See also
setDilation()
Deprecated Superseded by getDilationNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.49.3.3
getDilationNd()
Dims nvinfer1::IConvolutionLayer::getDilationNd ( ) const [inline], [noexcept]
Get the multi-dimension dilation of the convolution.
See also
setDilation()
206
Class Documentation
9.49.3.4
getKernelSize()
TRT DEPRECATED DimsHW nvinfer1::IConvolutionLayer::getKernelSize ( ) const [inline], [noexcept]
Get the HW kernel size of the convolution.
See also
setKernelSize()
Deprecated Superseded by getKernelSizeNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.49.3.5
getKernelSizeNd()
Dims nvinfer1::IConvolutionLayer::getKernelSizeNd ( ) const [inline], [noexcept]
Get the multi-dimension kernel size of the convolution.
See also
setKernelSizeNd()
9.49.3.6
getKernelWeights()
Weights nvinfer1::IConvolutionLayer::getKernelWeights ( ) const [inline], [noexcept]
Get the kernel weights of the convolution.
See also
setKernelWeights()
9.49.3.7
getNbGroups()
int32 t nvinfer1::IConvolutionLayer::getNbGroups ( ) const [inline], [noexcept]
Get the number of groups of the convolution.
See also
setNbGroups()
9.49 nvinfer1::IConvolutionLayer Class Reference
207
9.49.3.8
getNbOutputMaps()
int32 t nvinfer1::IConvolutionLayer::getNbOutputMaps ( ) const [inline], [noexcept]
Get the number of output maps for the convolution.
See also
setNbOutputMaps()
9.49.3.9
getPadding()
TRT DEPRECATED DimsHW nvinfer1::IConvolutionLayer::getPadding ( ) const [inline], [noexcept]
Get the padding of the convolution. If the padding is asymmetric, the pre-padding is returned.
See also
setPadding()
Deprecated Superseded by getPaddingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.49.3.10 getPaddingMode()
PaddingMode nvinfer1::IConvolutionLayer::getPaddingMode ( ) const [inline], [noexcept]
Get the padding mode.
Default: kEXPLICIT ROUND DOWN
See also
setPaddingMode()
208
Class Documentation
9.49.3.11 getPaddingNd()
Dims nvinfer1::IConvolutionLayer::getPaddingNd ( ) const [inline], [noexcept]
Get the multi-dimension padding of the convolution.
If the padding is asymmetric, the pre-padding is returned.
See also
setPaddingNd()
9.49.3.12 getPostPadding()
Dims nvinfer1::IConvolutionLayer::getPostPadding ( ) const [inline], [noexcept]
Get the post-padding.
See also
setPostPadding()
9.49.3.13 getPrePadding()
Dims nvinfer1::IConvolutionLayer::getPrePadding ( ) const [inline], [noexcept]
Get the pre-padding.
See also
setPrePadding()
9.49.3.14 getStride()
TRT DEPRECATED DimsHW nvinfer1::IConvolutionLayer::getStride ( ) const [inline], [noexcept]
Get the stride of the convolution.
Deprecated Superseded by getStrideNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.49 nvinfer1::IConvolutionLayer Class Reference
209
9.49.3.15 getStrideNd()
Dims nvinfer1::IConvolutionLayer::getStrideNd ( ) const [inline], [noexcept]
Get the multi-dimension stride of the convolution.
See also
setStrideNd()
9.49.3.16 setBiasWeights()
void nvinfer1::IConvolutionLayer::setBiasWeights (
Weights weights ) [inline], [noexcept]
Set the bias weights for the convolution.
Bias is optional. To omit bias, set the count value of the weights structure to zero.
The bias is applied per-channel, so the number of weights (if non-zero) must be equal to the number of output feature
maps.
See also
getBiasWeights()
9.49.3.17 setDilation()
TRT DEPRECATED void nvinfer1::IConvolutionLayer::setDilation (
DimsHW dilation ) [inline], [noexcept]
Set the dilation for a convolution.
Default: (1,1)
If executing this layer on DLA, both height and width must be in the range [1,32].
See also
getDilation()
Deprecated Superseded by setDilationNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
210
Class Documentation
9.49.3.18 setDilationNd()
void nvinfer1::IConvolutionLayer::setDilationNd (
Dims dilation ) [inline], [noexcept]
Set the multi-dimension dilation of the convolution.
Default: (1, 1, ..., 1)
If executing this layer on DLA, only support 2D padding, both height and width must be in the range [1,32].
See also
getDilation()
9.49.3.19 setInput()
void nvinfer1::ILayer::setInput (
int32 t index,
ITensor & tensor ) [inline], [noexcept]
Append or replace an input of this layer with a specific tensor.
Parameters
index
the index of the input to modify.
tensor
the new input tensor
The indices are as follows:
Input 0 is the input activation tensor. Input 1 is the kernel tensor. If used, the kernel weights parameter must be set to
empty weights. Input 2 is the bias tensor. If used, the bias parameter must be set to empty weights.
See also
getKernelWeights(), setKernelWeights(), getBiasWeights(), setBiasWeights()
9.49.3.20 setKernelSize()
TRT DEPRECATED void nvinfer1::IConvolutionLayer::setKernelSize (
DimsHW kernelSize ) [inline], [noexcept]
Set the HW kernel size of the convolution.
If executing this layer on DLA, both height and width of kernel size must be in the range [1,32].
9.49 nvinfer1::IConvolutionLayer Class Reference
211
See also
getKernelSize()
Deprecated Superseded by setKernelSizeNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.49.3.21 setKernelSizeNd()
void nvinfer1::IConvolutionLayer::setKernelSizeNd (
Dims kernelSize ) [inline], [noexcept]
Set the multi-dimension kernel size of the convolution.
If executing this layer on DLA, only support 2D kernel size, both height and width of kernel size must be in the range
[1,32].
See also
getKernelSizeNd()
9.49.3.22 setKernelWeights()
void nvinfer1::IConvolutionLayer::setKernelWeights (
Weights weights ) [inline], [noexcept]
Set the kernel weights for the convolution.
The weights are specified as a contiguous array in GKCRS order, where G is the number of groups, K the number of
output feature maps, C the number of input channels, and R and S are the height and width of the filter.
See also
getKernelWeights()
212
Class Documentation
9.49.3.23 setNbGroups()
void nvinfer1::IConvolutionLayer::setNbGroups (
int32 t nbGroups ) [inline], [noexcept]
Set the number of groups for a convolution.
The input tensor channels are divided into nbGroups groups, and a convolution is executed for each group, using a
filter per group. The results of the group convolutions are concatenated to form the output.
Note
When using groups in int8 mode, the size of the groups (i.e. the channel count divided by the group count) must
be a multiple of 4 for both input and output.
Default: 1
If executing this layer on DLA, the max number of groups is 8192.
See also
getNbGroups()
9.49.3.24 setNbOutputMaps()
void nvinfer1::IConvolutionLayer::setNbOutputMaps (
int32 t nbOutputMaps ) [inline], [noexcept]
Set the number of output maps for the convolution.
If executing this layer on DLA, the number of output maps must be in the range [1,8192].
See also
getNbOutputMaps()
9.49 nvinfer1::IConvolutionLayer Class Reference
213
9.49.3.25 setPadding()
TRT DEPRECATED void nvinfer1::IConvolutionLayer::setPadding (
DimsHW padding ) [inline], [noexcept]
Set the padding of the convolution.
The input will be zero-padded by this number of elements in the height and width directions. Padding is symmetric.
Default: (0,0)
If executing this layer on DLA, both height and width of padding must be in the range [0,31], and the padding size
must be less than the kernel size.
See also
getPadding()
Deprecated Superseded by setPaddingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.49.3.26 setPaddingMode()
void nvinfer1::IConvolutionLayer::setPaddingMode (
PaddingMode paddingMode ) [inline], [noexcept]
Set the padding mode.
Padding mode takes precedence if both setPaddingMode and setPre/PostPadding are used.
Default: kEXPLICIT ROUND DOWN
See also
getPaddingMode()
9.49.3.27 setPaddingNd()
void nvinfer1::IConvolutionLayer::setPaddingNd (
Dims padding ) [inline], [noexcept]
Set the multi-dimension padding of the convolution.
The input will be zero-padded by this number of elements in each dimension. Padding is symmetric.
Default: (0, 0, ..., 0)
If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range [0,31],
and the padding must be less than the kernel size.
See also
getPaddingNd() setPadding() getPadding()
214
Class Documentation
9.49.3.28 setPostPadding()
void nvinfer1::IConvolutionLayer::setPostPadding (
Dims padding ) [inline], [noexcept]
Set the multi-dimension post-padding of the convolution.
The end of the input will be zero-padded by this number of elements in each dimension.
Default: (0, 0, ..., 0)
If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range [0,31],
and the padding must be less than the kernel size.
See also
getPostPadding()
9.49.3.29 setPrePadding()
void nvinfer1::IConvolutionLayer::setPrePadding (
Dims padding ) [inline], [noexcept]
Set the multi-dimension pre-padding of the convolution.
The start of the input will be zero-padded by this number of elements in each dimension.
Default: (0, 0, ..., 0)
If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range [0,31],
and the padding must be less than the kernel size.
See also
getPrePadding()
9.49.3.30 setStride()
TRT DEPRECATED void nvinfer1::IConvolutionLayer::setStride (
DimsHW stride ) [inline], [noexcept]
Get the stride of the convolution.
Default: (1,1)
If executing this layer on DLA, both height and width of stride must be in the range [1,8].
See also
getStride()
Deprecated Superseded by setStrideNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.50 nvinfer1::ICudaEngine Class Reference
215
9.49.3.31 setStrideNd()
void nvinfer1::IConvolutionLayer::setStrideNd (
Dims stride ) [inline], [noexcept]
Set the multi-dimension stride of the convolution.
Default: (1, 1, ..., 1)
If executing this layer on DLA, only support 2D stride, both height and width of stride must be in the range [1,8].
See also
getStrideNd() setStride() getStride()
9.49.4
Member Data Documentation
9.49.4.1
mImpl
apiv::VConvolutionLayer∗ nvinfer1::IConvolutionLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.50
nvinfer1::ICudaEngine Class Reference
An engine for executing inference on a built network, with functionally unsafe features.
#include <NvInferRuntime.h>
Inheritance diagram for nvinfer1::ICudaEngine:
nvinfer1::INoCopy
nvinfer1::ICudaEngine
216
Class Documentation
Public Member Functions
•
virtual ∼ICudaEngine () noexcept=default
•
TRT DEPRECATED int32 t getNbBindings () const noexcept
Get the number of binding indices.
•
TRT DEPRECATED int32 t getBindingIndex (char const ∗name) const noexcept
Retrieve the binding index for a named tensor.
•
TRT DEPRECATED char const ∗ getBindingName (int32 t bindingIndex) const noexcept
Retrieve the name corresponding to a binding index.
•
TRT DEPRECATED bool bindingIsInput (int32 t bindingIndex) const noexcept
Determine whether a binding is an input binding.
•
TRT DEPRECATED Dims getBindingDimensions (int32 t bindingIndex) const noexcept
Get the dimensions of a binding.
•
Dims getTensorShape (char const ∗tensorName) const noexcept
Get shape of an input or output tensor.
•
TRT DEPRECATED DataType getBindingDataType (int32 t bindingIndex) const noexcept
Determine the required data type for a buffer from its binding index.
•
DataType getTensorDataType (char const ∗tensorName) const noexcept
Determine the required data type for a buffer from its tensor name.
•
TRT DEPRECATED int32 t getMaxBatchSize () const noexcept
Get the maximum batch size which can be used for inference. Should only be called if the engine is built from an
INetworkDefinition with implicit batch dimension mode.
•
int32 t getNbLayers () const noexcept
Get the number of layers in the network.
•
IHostMemory ∗ serialize () const noexcept
Serialize the network to a stream.
•
IExecutionContext ∗ createExecutionContext () noexcept
Create an execution context.
•
TRT DEPRECATED void destroy () noexcept
Destroy this object;.
•
TRT DEPRECATED TensorLocation getLocation (int32 t bindingIndex) const noexcept
Get location of binding.
•
TensorLocation getTensorLocation (char const ∗tensorName) const noexcept
Get whether an input or output tensor must be on GPU or CPU.
•
bool isShapeInferenceIO (char const ∗tensorName) const noexcept
True if tensor is required as input for shape calculations or is output from shape calculations.
•
TensorIOMode getTensorIOMode (char const ∗tensorName) const noexcept
Determine whether a tensor is an input or output tensor.
•
IExecutionContext ∗ createExecutionContextWithoutDeviceMemory () noexcept
create an execution context without any device memory allocated
•
size t getDeviceMemorySize () const noexcept
Return the amount of device memory required by an execution context.
•
bool isRefittable () const noexcept
Return true if an engine can be refit.
•
TRT DEPRECATED int32 t getBindingBytesPerComponent (int32 t bindingIndex) const noexcept
Return the number of bytes per component of an element.
•
int32 t getTensorBytesPerComponent (char const ∗tensorName) const noexcept
9.50 nvinfer1::ICudaEngine Class Reference
217
Return the number of bytes per component of an element, or -1 if the provided name does not map to an input or output
tensor.
•
int32 t getTensorBytesPerComponent (char const ∗tensorName, int32 t profileIndex) const noexcept
Return the number of bytes per component of an element of given profile, or -1 if the provided name does not map to an
input or output tensor.
•
TRT DEPRECATED int32 t getBindingComponentsPerElement (int32 t bindingIndex) const noexcept
Return the number of components included in one element.
•
int32 t getTensorComponentsPerElement (char const ∗tensorName) const noexcept
Return the number of components included in one element, or -1 if the provided name does not map to an input or output
tensor.
•
int32 t getTensorComponentsPerElement (char const ∗tensorName, int32 t profileIndex) const noexcept
Return the number of components included in one element of given profile, or -1 if the provided name does not map to an
input or output tensor.
•
TRT DEPRECATED TensorFormat getBindingFormat (int32 t bindingIndex) const noexcept
Return the binding format.
•
TensorFormat getTensorFormat (char const ∗tensorName) const noexcept
Return the tensor format, or TensorFormat::kLINEAR if the provided name does not map to an input or output tensor.
•
TensorFormat getTensorFormat (char const ∗tensorName, int32 t profileIndex) const noexcept
Return the tensor format of given profile, or TensorFormat::kLINEAR if the provided name does not map to an input or
output tensor.
•
TRT DEPRECATED char const ∗ getBindingFormatDesc (int32 t bindingIndex) const noexcept
Return the human readable description of the tensor format, or nullptr if the provided name does not map to an input or
output tensor.
•
char const ∗ getTensorFormatDesc (char const ∗tensorName) const noexcept
Return the human readable description of the tensor format, or empty string if the provided name does not map to an
input or output tensor.
•
char const ∗ getTensorFormatDesc (char const ∗tensorName, int32 t profileIndex) const noexcept
Return the human readable description of the tensor format of given profile, or empty string if the provided name does
not map to an input or output tensor.
•
TRT DEPRECATED int32 t getBindingVectorizedDim (int32 t bindingIndex) const noexcept
Return the dimension index that the buffer is vectorized, or -1 is the name is not found.
•
int32 t getTensorVectorizedDim (char const ∗tensorName) const noexcept
Return the dimension index that the buffer is vectorized, or -1 if the provided name does not map to an input or output
tensor.
•
int32 t getTensorVectorizedDim (char const ∗tensorName, int32 t profileIndex) const noexcept
Return the dimension index that the buffer is vectorized of given profile, or -1 if the provided name does not map to an
input or output tensor.
•
char const ∗ getName () const noexcept
Returns the name of the network associated with the engine.
•
int32 t getNbOptimizationProfiles () const noexcept
Get the number of optimization profiles defined for this engine.
•
TRT DEPRECATED Dims getProfileDimensions (int32 t bindingIndex, int32 t profileIndex, OptProfileSelector
select) const noexcept
Get the minimum / optimum / maximum dimensions for a particular input binding under an optimization profile.
•
Dims getProfileShape (char const ∗tensorName, int32 t profileIndex, OptProfileSelector select) const noexcept
Get the minimum / optimum / maximum dimensions for an input tensor given its name under an optimization profile.
•
TRT DEPRECATED int32 t const ∗ getProfileShapeValues (int32 t profileIndex, int32 t inputIndex,
OptProfileSelector select) const noexcept
Get minimum / optimum / maximum values for an input shape binding under an optimization profile.
218
Class Documentation
•
TRT DEPRECATED bool isShapeBinding (int32 t bindingIndex) const noexcept
True if tensor is required as input for shape calculations or output from them.
•
TRT DEPRECATED bool isExecutionBinding (int32 t bindingIndex) const noexcept
True if pointer to tensor data is required for execution phase, false if nullptr can be supplied.
•
EngineCapability getEngineCapability () const noexcept
Determine what execution capability this engine has.
•
void setErrorRecorder (IErrorRecorder ∗recorder) noexcept
Set the ErrorRecorder for this interface.
•
IErrorRecorder ∗ getErrorRecorder () const noexcept
Get the ErrorRecorder assigned to this interface.
•
bool hasImplicitBatchDimension () const noexcept
Query whether the engine was built with an implicit batch dimension.
•
TacticSources getTacticSources () const noexcept
return the tactic sources required by this engine.
•
ProfilingVerbosity getProfilingVerbosity () const noexcept
Return the ProfilingVerbosity the builder config was set to when the engine was built.
•
IEngineInspector ∗ createEngineInspector () const noexcept
Create a new engine inspector which prints the layer information in an engine or an execution context.
•
int32 t getNbIOTensors () const noexcept
Return number of IO tensors.
•
char const ∗ getIOTensorName (int32 t index) const noexcept
Return name of an IO tensor.
•
HardwareCompatibilityLevel getHardwareCompatibilityLevel () const noexcept
Return the hardware compatibility level of this engine.
•
int32 t getNbAuxStreams () const noexcept
Return the number of auxiliary streams used by this engine.
Protected Attributes
• apiv::VCudaEngine ∗ mImpl
Additional Inherited Members
9.50.1
Detailed Description
An engine for executing inference on a built network, with functionally unsafe features.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.50.2
Constructor & Destructor Documentation
9.50 nvinfer1::ICudaEngine Class Reference
219
9.50.2.1
∼ICudaEngine()
virtual nvinfer1::ICudaEngine::∼ICudaEngine ( ) [virtual], [default], [noexcept]
9.50.3
Member Function Documentation
9.50.3.1
bindingIsInput()
TRT DEPRECATED bool nvinfer1::ICudaEngine::bindingIsInput (
int32 t bindingIndex ) const [inline], [noexcept]
Determine whether a binding is an input binding.
Parameters
bindingIndex
The binding index.
Returns
True if the index corresponds to an input binding and the index is in range.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorIOMode().
See also
getTensorIOMode()
9.50.3.2
createEngineInspector()
IEngineInspector ∗ nvinfer1::ICudaEngine::createEngineInspector ( ) const [inline], [noexcept]
Create a new engine inspector which prints the layer information in an engine or an execution context.
See also
IEngineInspector.
220
Class Documentation
9.50.3.3
createExecutionContext()
IExecutionContext ∗ nvinfer1::ICudaEngine::createExecutionContext ( ) [inline], [noexcept]
Create an execution context.
The execution context created will call setOptimizationProfile(0) implicitly if there are no other execution contexts
assigned to optimization profile 0. This functionality is deprecated in TensorRT 8.6 and will instead default all opti-
mization profiles to 0 starting in TensorRT 9.0. If an error recorder has been set for the engine, it will also be passed to
the execution context.
See also
IExecutionContext.
IExecutionContext::setOptimizationProfile()
9.50.3.4
createExecutionContextWithoutDeviceMemory()
IExecutionContext ∗ nvinfer1::ICudaEngine::createExecutionContextWithoutDeviceMemory ( ) [inline],
[noexcept]
create an execution context without any device memory allocated
The memory for execution of this device context must be supplied by the application.
9.50.3.5
destroy()
TRT DEPRECATED void nvinfer1::ICudaEngine::destroy ( ) [inline], [noexcept]
Destroy this object;.
Deprecated Deprecated in TRT 8.0. Superseded by delete.
Warning
Calling destroy on a managed pointer will result in a double-free error.
9.50.3.6
getBindingBytesPerComponent()
TRT DEPRECATED int32 t nvinfer1::ICudaEngine::getBindingBytesPerComponent (
int32 t bindingIndex ) const [inline], [noexcept]
Return the number of bytes per component of an element.
The vector component size is returned if getBindingVectorizedDim() != -1.
9.50 nvinfer1::ICudaEngine Class Reference
221
Parameters
bindingIndex
The binding Index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorBytesPerComponent().
See also
getBindingVectorizedDim()
getTensorBytesPerComponent()
9.50.3.7
getBindingComponentsPerElement()
TRT DEPRECATED int32 t nvinfer1::ICudaEngine::getBindingComponentsPerElement (
int32 t bindingIndex ) const [inline], [noexcept]
Return the number of components included in one element.
The number of elements in the vectors is returned if getBindingVectorizedDim() != -1.
Parameters
bindingIndex
The binding Index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorComponentsPerElement().
See also
getBindingVectorizedDim()
9.50.3.8
getBindingDataType()
TRT DEPRECATED DataType nvinfer1::ICudaEngine::getBindingDataType (
int32 t bindingIndex ) const [inline], [noexcept]
Determine the required data type for a buffer from its binding index.
222
Class Documentation
Parameters
bindingIndex
The binding index.
Returns
The type of the data in the buffer.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorDataType().
See also
getTensorDataType()
9.50.3.9
getBindingDimensions()
TRT DEPRECATED Dims nvinfer1::ICudaEngine::getBindingDimensions (
int32 t bindingIndex ) const [inline], [noexcept]
Get the dimensions of a binding.
Parameters
bindingIndex
The binding index.
Returns
The dimensions of the binding if the index is in range, otherwise Dims(). Has -1 for any dimension that varies
within the optimization profile.
For example, suppose an INetworkDefinition has an input with shape [-1,-1] that becomes a binding b in the engine. If
the associated optimization profile specifies that b has minimum dimensions as [6,9] and maximum dimensions [7,9],
getBindingDimensions(b) returns [-1,9], despite the second dimension being dynamic in the INetworkDefinition.
Because each optimization profile has separate bindings, the returned value can differ across profiles. Consider another
binding b' for the same network input, but for another optimization profile. If that other profile specifies minimum
dimensions [5,8] and maximum dimensions [5,9], getBindingDimensions(b') returns [5,-1].
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorShape().
See also
getTensorShape()
9.50 nvinfer1::ICudaEngine Class Reference
223
9.50.3.10 getBindingFormat()
TRT DEPRECATED TensorFormat nvinfer1::ICudaEngine::getBindingFormat (
int32 t bindingIndex ) const [inline], [noexcept]
Return the binding format.
Parameters
bindingIndex
The binding Index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorFormat().
See also
getTensorFormat()
9.50.3.11 getBindingFormatDesc()
TRT DEPRECATED char const ∗ nvinfer1::ICudaEngine::getBindingFormatDesc (
int32 t bindingIndex ) const [inline], [noexcept]
Return the human readable description of the tensor format, or nullptr if the provided name does not map to an input
or output tensor.
The description includes the order, vectorization, data type, and strides. Examples are shown as follows: Example 1:
kCHW + FP32 ”Row major linear FP32 format” Example 2: kCHW2 + FP16 ”Two wide channel vectorized row major
FP16 format” Example 3: kHWC8 + FP16 + Line Stride = 32 ”Channel major FP16 format where C % 8 == 0 and H
Stride % 32 == 0”
Parameters
bindingIndex
The binding Index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorFormatDesc().
See also
getTensorFormatDesc()
224
Class Documentation
9.50.3.12 getBindingIndex()
TRT DEPRECATED int32 t nvinfer1::ICudaEngine::getBindingIndex (
char const ∗ name ) const [inline], [noexcept]
Retrieve the binding index for a named tensor.
IExecutionContext::enqueueV2() and IExecutionContext::executeV2() require an array of buffers.
Engine bindings map from tensor names to indices in this array. Binding indices are assigned at engine build time, and
take values in the range [0 ... n-1] where n is the total number of inputs and outputs.
To get the binding index of the name in an optimization profile with index k > 0, mangle the name by appending ”
[profile k]”, as described for method getBindingName().
Parameters
name
The tensor name.
Returns
The binding index for the named tensor, or -1 if the provided name does not map to an input or output tensor.
Warning
The string name must be null-terminated, and be at most 4096 bytes including the terminator.
Deprecated Deprecated in TensorRT 8.5. Superseded by name-based methods. Use them instead of binding-index
based methods.
See also
getNbBindings() getBindingName()
9.50.3.13 getBindingName()
TRT DEPRECATED char const ∗ nvinfer1::ICudaEngine::getBindingName (
int32 t bindingIndex ) const [inline], [noexcept]
Retrieve the name corresponding to a binding index.
This is the reverse mapping to that provided by getBindingIndex().
For optimization profiles with an index k > 0, the name is mangled by appending ” [profile k]”, with k written in
decimal. For example, if the tensor in the INetworkDefinition had the name ”foo”, and bindingIndex refers to that
tensor in the optimization profile with index 3, getBindingName returns ”foo [profile 3]”.
9.50 nvinfer1::ICudaEngine Class Reference
225
Parameters
bindingIndex
The binding index.
Returns
The name corresponding to the index, or nullptr if the index is out of range.
Deprecated Deprecated in TensorRT 8.5. Superseded by name-based methods. Use them instead of binding-index
based methods.
See also
getBindingIndex()
9.50.3.14 getBindingVectorizedDim()
TRT DEPRECATED int32 t nvinfer1::ICudaEngine::getBindingVectorizedDim (
int32 t bindingIndex ) const [inline], [noexcept]
Return the dimension index that the buffer is vectorized, or -1 is the name is not found.
Specifically -1 is returned if scalars per vector is 1.
Parameters
bindingIndex
The binding Index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorVectorizedDim().
See also
getTensorVectorizedDim()
9.50.3.15 getDeviceMemorySize()
size t nvinfer1::ICudaEngine::getDeviceMemorySize ( ) const [inline], [noexcept]
Return the amount of device memory required by an execution context.
See also
IExecutionContext::setDeviceMemory()
226
Class Documentation
9.50.3.16 getEngineCapability()
EngineCapability nvinfer1::ICudaEngine::getEngineCapability ( ) const [inline], [noexcept]
Determine what execution capability this engine has.
If the engine has EngineCapability::kSTANDARD, then all engine functionality is valid.
If the engine
has EngineCapability::kSAFETY, then only the functionality in safe engine is valid.
If the engine has
EngineCapability::kDLA STANDALONE, then only serialize, destroy, and const-accessor functions are valid.
Returns
The EngineCapability flag that the engine was built for.
9.50.3.17 getErrorRecorder()
IErrorRecorder ∗ nvinfer1::ICudaEngine::getErrorRecorder ( ) const [inline], [noexcept]
Get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A nullptr will be returned if an error handler has not
been set.
Returns
A pointer to the IErrorRecorder object that has been registered.
See also
setErrorRecorder()
9.50.3.18 getHardwareCompatibilityLevel()
HardwareCompatibilityLevel nvinfer1::ICudaEngine::getHardwareCompatibilityLevel ( ) const [inline],
[noexcept]
Return the hardware compatibility level of this engine.
Returns
hardwareCompatibilityLevel The level of hardware compatibility.
This is only supported for Ampere and newer architectures.
9.50.3.19 getIOTensorName()
char const ∗ nvinfer1::ICudaEngine::getIOTensorName (
int32 t index ) const [inline], [noexcept]
Return name of an IO tensor.
9.50 nvinfer1::ICudaEngine Class Reference
227
Parameters
index
value between 0 and getNbIOTensors()-1
See also
getNbIOTensors()
9.50.3.20 getLocation()
TRT DEPRECATED TensorLocation nvinfer1::ICudaEngine::getLocation (
int32 t bindingIndex ) const [inline], [noexcept]
Get location of binding.
This lets you know whether the binding should be a pointer to device or host memory.
Parameters
bindingIndex
The binding index.
Returns
The location of the bound tensor with given index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorLocation().
See also
ITensor::setLocation() ITensor::getLocation()
getTensorLocation()
9.50.3.21 getMaxBatchSize()
TRT DEPRECATED int32 t nvinfer1::ICudaEngine::getMaxBatchSize ( ) const [inline], [noexcept]
Get the maximum batch size which can be used for inference. Should only be called if the engine is built from an
INetworkDefinition with implicit batch dimension mode.
Returns
The maximum batch size for this engine.
228
Class Documentation
Warning
For an engine built from an INetworkDefinition with explicit batch dimension mode, this will always return
1.
Deprecated Deprecated in TensorRT 8.4.
9.50.3.22 getName()
char const ∗ nvinfer1::ICudaEngine::getName ( ) const [inline], [noexcept]
Returns the name of the network associated with the engine.
The name is set during network creation and is retrieved after building or deserialization.
See also
INetworkDefinition::setName(), INetworkDefinition::getName()
Returns
A null-terminated C-style string representing the name of the network.
9.50.3.23 getNbAuxStreams()
int32 t nvinfer1::ICudaEngine::getNbAuxStreams ( ) const [inline], [noexcept]
Return the number of auxiliary streams used by this engine.
This number will be less than or equal to the maximum allowed number of auxiliary streams set by
IBuilderConfig::setMaxAuxStreams() API call when the engine was built.
Returns
the number of auxiliary streams used by this engine.
See also
IBuilderConfig::setMaxAuxStreams(), IExecutionContext::setAuxStreams()
9.50 nvinfer1::ICudaEngine Class Reference
229
9.50.3.24 getNbBindings()
TRT DEPRECATED int32 t nvinfer1::ICudaEngine::getNbBindings ( ) const [inline], [noexcept]
Get the number of binding indices.
There are separate binding indices for each optimization profile. This method returns the total over all profiles. If the
engine has been built for K profiles, the first getNbBindings() / K bindings are used by profile number 0, the following
getNbBindings() / K bindings are used by profile number 1 etc.
Deprecated Deprecated in TensorRT 8.5. Superseded by getNbIOTensors.
See also
getBindingIndex()
9.50.3.25 getNbIOTensors()
int32 t nvinfer1::ICudaEngine::getNbIOTensors ( ) const [inline], [noexcept]
Return number of IO tensors.
It is the number of input and output tensors for the network from which the engine was built. The names of the IO
tensors can be discovered by calling getIOTensorName(i) for i in 0 to getNbIOTensors()-1.
See also
getIOTensorName()
9.50.3.26 getNbLayers()
int32 t nvinfer1::ICudaEngine::getNbLayers ( ) const [inline], [noexcept]
Get the number of layers in the network.
The number of layers in the network is not necessarily the number in the original network definition, as layers may be
combined or eliminated as the engine is optimized. This value can be useful when building per-layer tables, such as
when aggregating profiling data over a number of executions.
Returns
The number of layers in the network.
230
Class Documentation
9.50.3.27 getNbOptimizationProfiles()
int32 t nvinfer1::ICudaEngine::getNbOptimizationProfiles ( ) const [inline], [noexcept]
Get the number of optimization profiles defined for this engine.
Returns
Number of optimization profiles. It is always at least 1.
See also
IExecutionContext::setOptimizationProfile()
9.50.3.28 getProfileDimensions()
TRT DEPRECATED Dims nvinfer1::ICudaEngine::getProfileDimensions (
int32 t bindingIndex,
int32 t profileIndex,
OptProfileSelector select ) const [inline], [noexcept]
Get the minimum / optimum / maximum dimensions for a particular input binding under an optimization profile.
Parameters
bindingIndex
The input binding index, which must belong to the given profile, or be between 0 and
bindingsPerProfile-1 as described below.
profileIndex
The profile index, which must be between 0 and getNbOptimizationProfiles()-1.
select
Whether to query the minimum, optimum, or maximum dimensions for this binding.
Returns
The minimum / optimum / maximum dimensions for this binding in this profile. If the profileIndex or binding←↩
Index are invalid, return Dims with nbDims=-1.
For backwards compatibility with earlier versions of TensorRT, if the bindingIndex does not belong to the current
optimization profile, but is between 0 and bindingsPerProfile-1, where bindingsPerProfile = getNbBindings()/getNb←↩
OptimizationProfiles, then a corrected bindingIndex is used instead, computed by:
profileIndex * bindingsPerProfile + bindingIndex % bindingsPerProfile
Otherwise the bindingIndex is considered invalid.
Deprecated Deprecated in TensorRT 8.5. Superseded by getProfileShape().
9.50 nvinfer1::ICudaEngine Class Reference
231
See also
getProfileShape()
9.50.3.29 getProfileShape()
Dims nvinfer1::ICudaEngine::getProfileShape (
char const ∗ tensorName,
int32 t profileIndex,
OptProfileSelector select ) const [inline], [noexcept]
Get the minimum / optimum / maximum dimensions for an input tensor given its name under an optimization profile.
Parameters
tensorName
The name of an input tensor.
profileIndex
The profile index, which must be between 0 and getNbOptimizationProfiles()-1.
select
Whether to query the minimum, optimum, or maximum dimensions for this input tensor.
Returns
The minimum / optimum / maximum dimensions for an input tensor in this profile. If the profileIndex is invalid
or provided name does not map to an input tensor, return Dims{-1, {}}
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.50.3.30 getProfileShapeValues()
TRT DEPRECATED int32 t const ∗ nvinfer1::ICudaEngine::getProfileShapeValues (
int32 t profileIndex,
int32 t inputIndex,
OptProfileSelector select ) const [inline], [noexcept]
Get minimum / optimum / maximum values for an input shape binding under an optimization profile.
Parameters
profileIndex
The profile index (must be between 0 and getNbOptimizationProfiles()-1)
inputIndex
The input index (must be between 0 and getNbBindings() - 1)
select
Whether to query the minimum, optimum, or maximum shape values for this binding.
232
Class Documentation
Returns
If the binding is an input shape binding, return a pointer to an array that has the same number of elements as the
corresponding tensor, i.e. 1 if dims.nbDims == 0, or dims.d[0] if dims.nbDims == 1, where dims = getBinding←↩
Dimensions(inputIndex). The array contains the elementwise minimum / optimum / maximum values for this
shape binding under the profile. If either of the indices is out of range, or if the binding is not an input shape
binding, return nullptr.
For backwards compatibility with earlier versions of TensorRT, a bindingIndex that does not belong to the profile is
corrected as described for getProfileDimensions().
Deprecated Deprecated in TensorRT 8.5. Superseded by getShapeValues(). Difference between Execution and shape
tensor is superficial since TensorRT 8.5.
See also
getProfileDimensions() getShapeValues()
9.50.3.31 getProfilingVerbosity()
ProfilingVerbosity nvinfer1::ICudaEngine::getProfilingVerbosity ( ) const [inline], [noexcept]
Return the ProfilingVerbosity the builder config was set to when the engine was built.
Returns
the profiling verbosity the builder config was set to when the engine was built.
See also
IBuilderConfig::setProfilingVerbosity()
9.50.3.32 getTacticSources()
TacticSources nvinfer1::ICudaEngine::getTacticSources ( ) const [inline], [noexcept]
return the tactic sources required by this engine.
The value returned is equal to zero or more tactics sources set at build time via setTacticSources() in IBuilderConfig.
Sources set by the latter but not returned by ICudaEngine::getTacticSources do not reduce overall engine execution
time, and can be removed from future builds to reduce build time.
See also
IBuilderConfig::setTacticSources()
9.50 nvinfer1::ICudaEngine Class Reference
233
9.50.3.33 getTensorBytesPerComponent() [1/2]
int32 t nvinfer1::ICudaEngine::getTensorBytesPerComponent (
char const ∗ tensorName ) const [inline], [noexcept]
Return the number of bytes per component of an element, or -1 if the provided name does not map to an input or output
tensor.
The vector component size is returned if getTensorVectorizedDim() != -1.
234
Class Documentation
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
The function can only return the result of profile 0, and issues a warning message when there are multiple
profiles in the engine, use getTensorBytesPerComponent with profileIndex when there are multiple profiles.
See also
getTensorVectorizedDim()
getTensorBytesPerComponent(tensorName, profileIndex)
9.50.3.34 getTensorBytesPerComponent() [2/2]
int32 t nvinfer1::ICudaEngine::getTensorBytesPerComponent (
char const ∗ tensorName,
int32 t profileIndex ) const [inline], [noexcept]
Return the number of bytes per component of an element of given profile, or -1 if the provided name does not map to
an input or output tensor.
The vector component size is returned if getTensorVectorizedDim(tensorName, profileIndex) != -1.
Parameters
tensorName
The name of an input or output tensor.
profileIndex
The profile index to query
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
See also
getTensorVectorizedDim(tensorName, profileIndex)
9.50 nvinfer1::ICudaEngine Class Reference
235
9.50.3.35 getTensorComponentsPerElement() [1/2]
int32 t nvinfer1::ICudaEngine::getTensorComponentsPerElement (
char const ∗ tensorName ) const [inline], [noexcept]
Return the number of components included in one element, or -1 if the provided name does not map to an input or
output tensor.
The number of elements in the vectors is returned if getTensorVectorizedDim() != -1.
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
The function can only return the result of profile 0, and issues a warning message when there are multi-
ple profiles in the engine, use getTensorComponentsPerElement with profileIndex when there are multiple
profiles.
See also
getTensorVectorizedDim()
getTensorComponentsPerElement(tensorName, profileIndex)
9.50.3.36 getTensorComponentsPerElement() [2/2]
int32 t nvinfer1::ICudaEngine::getTensorComponentsPerElement (
char const ∗ tensorName,
int32 t profileIndex ) const [inline], [noexcept]
Return the number of components included in one element of given profile, or -1 if the provided name does not map to
an input or output tensor.
The number of elements in the vectors is returned if getTensorVectorizedDim(tensorName, profileIndex) != -1.
Parameters
tensorName
The name of an input or output tensor.
profileIndex
The profile index to query
236
Class Documentation
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
See also
getTensorVectorizedDim(tensorName, profileIndex)
9.50.3.37 getTensorDataType()
DataType nvinfer1::ICudaEngine::getTensorDataType (
char const ∗ tensorName ) const [inline], [noexcept]
Determine the required data type for a buffer from its tensor name.
Parameters
tensorName
The name of an input or output tensor.
Returns
The type of the data in the buffer, or DataType::kFLOAT if the provided name does not map to an input or output
tensor.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.50.3.38 getTensorFormat() [1/2]
TensorFormat nvinfer1::ICudaEngine::getTensorFormat (
char const ∗ tensorName ) const [inline], [noexcept]
Return the tensor format, or TensorFormat::kLINEAR if the provided name does not map to an input or output tensor.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
This API can only return the tensor format of profile 0, and issues a warning message when there are multiple
profiles in the engine, use getTensorFormat with profileIndex when there are multiple profiles.
See also
getTensorFormat(tensorName, profileIndex)
9.50 nvinfer1::ICudaEngine Class Reference
237
9.50.3.39 getTensorFormat() [2/2]
TensorFormat nvinfer1::ICudaEngine::getTensorFormat (
char const ∗ tensorName,
int32 t profileIndex ) const [inline], [noexcept]
Return the tensor format of given profile, or TensorFormat::kLINEAR if the provided name does not map to an input
or output tensor.
Parameters
tensorName
The name of an input or output tensor.
profileIndex
The profile index to query the format for.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.50.3.40 getTensorFormatDesc() [1/2]
char const ∗ nvinfer1::ICudaEngine::getTensorFormatDesc (
char const ∗ tensorName ) const [inline], [noexcept]
Return the human readable description of the tensor format, or empty string if the provided name does not map to an
input or output tensor.
The description includes the order, vectorization, data type, and strides. Examples are shown as follows: Example 1:
kCHW + FP32 ”Row major linear FP32 format” Example 2: kCHW2 + FP16 ”Two wide channel vectorized row major
FP16 format” Example 3: kHWC8 + FP16 + Line Stride = 32 ”Channel major FP16 format where C % 8 == 0 and H
Stride % 32 == 0”
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
The function can only return the result of profile 0, and issues a warning message when there are multiple
profiles in the engine, use getTensorFormatDesc with profileIndex when there are multiple profiles.
9.50.3.41 getTensorFormatDesc() [2/2]
char const ∗ nvinfer1::ICudaEngine::getTensorFormatDesc (
238
Class Documentation
char const ∗ tensorName,
int32 t profileIndex ) const [inline], [noexcept]
Return the human readable description of the tensor format of given profile, or empty string if the provided name does
not map to an input or output tensor.
The description includes the order, vectorization, data type, and strides. Examples are shown as follows: Example 1:
kCHW + FP32 ”Row major linear FP32 format” Example 2: kCHW2 + FP16 ”Two wide channel vectorized row major
FP16 format” Example 3: kHWC8 + FP16 + Line Stride = 32 ”Channel major FP16 format where C % 8 == 0 and H
Stride % 32 == 0”
Parameters
tensorName
The name of an input or output tensor.
profileIndex
The profile index to query the format for.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.50.3.42 getTensorIOMode()
TensorIOMode nvinfer1::ICudaEngine::getTensorIOMode (
char const ∗ tensorName ) const [inline], [noexcept]
Determine whether a tensor is an input or output tensor.
Parameters
tensorName
The name of an input or output tensor.
Returns
kINPUT if tensorName is an input, kOUTPUT if tensorName is an output, or kNONE if neither.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.50.3.43 getTensorLocation()
TensorLocation nvinfer1::ICudaEngine::getTensorLocation (
char const ∗ tensorName ) const [inline], [noexcept]
Get whether an input or output tensor must be on GPU or CPU.
9.50 nvinfer1::ICudaEngine Class Reference
239
Parameters
tensorName
The name of an input or output tensor.
Returns
TensorLocation::kDEVICE if tensorName must be on GPU, or TensorLocation::kHOST if on CPU, or
TensorLocation::kDEVICE if the provided name does not map to an input or output tensor.
The location is established at build time. E.g. shape tensors inputs are typically required to be on the CPU.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.50.3.44 getTensorShape()
Dims nvinfer1::ICudaEngine::getTensorShape (
char const ∗ tensorName ) const [inline], [noexcept]
Get shape of an input or output tensor.
Parameters
tensorName
The name of an input or output tensor.
Returns
shape of the tensor, with -1 in place of each dynamic runtime dimension, or Dims{-1, {}} if the provided name
does not map to an input or output tensor.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.50.3.45 getTensorVectorizedDim() [1/2]
int32 t nvinfer1::ICudaEngine::getTensorVectorizedDim (
char const ∗ tensorName ) const [inline], [noexcept]
Return the dimension index that the buffer is vectorized, or -1 if the provided name does not map to an input or output
tensor.
Specifically -1 is returned if scalars per vector is 1.
240
Class Documentation
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
The function can only return the result of profile 0, and issues a warning message when there are multiple
profiles in the engine, use getTensorVectorizedDim with profileIndex when there are multiple profiles.
9.50.3.46 getTensorVectorizedDim() [2/2]
int32 t nvinfer1::ICudaEngine::getTensorVectorizedDim (
char const ∗ tensorName,
int32 t profileIndex ) const [inline], [noexcept]
Return the dimension index that the buffer is vectorized of given profile, or -1 if the provided name does not map to an
input or output tensor.
Specifically -1 is returned if scalars per vector is 1.
Parameters
tensorName
The name of an input.
profileIndex
The profile index to query the format for.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.50.3.47 hasImplicitBatchDimension()
bool nvinfer1::ICudaEngine::hasImplicitBatchDimension ( ) const [inline], [noexcept]
Query whether the engine was built with an implicit batch dimension.
Returns
True if tensors have implicit batch dimension, false otherwise.
This is an engine-wide property. Either all tensors in the engine have an implicit batch dimension or none of them do.
hasImplicitBatchDimension() is true if and only if the INetworkDefinition from which this engine was built was created
with createNetworkV2() without NetworkDefinitionCreationFlag::kEXPLICIT BATCH flag.
9.50 nvinfer1::ICudaEngine Class Reference
241
See also
createNetworkV2
9.50.3.48 isExecutionBinding()
TRT DEPRECATED bool nvinfer1::ICudaEngine::isExecutionBinding (
int32 t bindingIndex ) const [inline], [noexcept]
True if pointer to tensor data is required for execution phase, false if nullptr can be supplied.
For example, if a network uses an input tensor with binding i ONLY as the ”reshape dimensions” input of IShuffleLayer,
then isExecutionBinding(i) is false, and a nullptr can be supplied for it when calling IExecutionContext::execute or
IExecutionContext::enqueue.
Deprecated No name-based equivalent replacement. Use getTensorLocation() instead to know the location of tensor
data. Distinction between execution binding and shape binding is superficial since TensorRT 8.5.
See also
isShapeBinding() getTensorLocation()
9.50.3.49 isRefittable()
bool nvinfer1::ICudaEngine::isRefittable ( ) const [inline], [noexcept]
Return true if an engine can be refit.
See also
nvinfer1::createInferRefitter()
242
Class Documentation
9.50.3.50 isShapeBinding()
TRT DEPRECATED bool nvinfer1::ICudaEngine::isShapeBinding (
int32 t bindingIndex ) const [inline], [noexcept]
True if tensor is required as input for shape calculations or output from them.
TensorRT evaluates a network in two phases:
1. Compute shape information required to determine memory allocation requirements and validate that runtime
sizes make sense.
2. Process tensors on the device.
Some tensors are required in phase 1. These tensors are called ”shape tensors”, and always have type Int32 and no
more than one dimension. These tensors are not always shapes themselves, but might be used to calculate tensor shapes
for phase 2.
isShapeBinding(i) returns true if the tensor is a required input or an output computed in phase 1. isExecutionBinding(i)
returns true if the tensor is a required input or an output computed in phase 2.
For example, if a network uses an input tensor with binding i as an addend to an IElementWiseLayer that computes the
”reshape dimensions” for IShuffleLayer, then isShapeBinding(i) == true.
It's possible to have a tensor be required by both phases. For instance, a tensor can be used for the ”reshape dimensions”
and as the indices for an IGatherLayer collecting floating-point data.
It's also possible to have a tensor be required by neither phase, but nonetheless shows up in the engine's inputs. For
example, if an input tensor is used only as an input to IShapeLayer, only its shape matters and its values are irrelevant.
Deprecated Use name-based isShapeInferenceIO() instead to know whether a tensor is a shape tensor.
See also
isExecutionBinding() isShapeInferenceIO()
9.50.3.51 isShapeInferenceIO()
bool nvinfer1::ICudaEngine::isShapeInferenceIO (
char const ∗ tensorName ) const [inline], [noexcept]
True if tensor is required as input for shape calculations or is output from shape calculations.
Return true for either of the following conditions:
• The tensor is a network input, and its value is required for IExecutionContext::getTensorShape() to return the
shape of a network output.
• The tensor is a network output, and inferShape() will compute its values.
For example, if a network uses an input tensor ”foo” as an addend to an IElementWiseLayer that computes the ”reshape
dimensions” for IShuffleLayer, then isShapeInferenceIO(”foo”) == true. If the network copies said input tensor ”foo”
to an output ”bar”, then isShapeInferenceIO(”bar”) == true and IExecutionContext::inferShapes() will write to ”bar”.
9.50 nvinfer1::ICudaEngine Class Reference
243
9.50.3.52 serialize()
IHostMemory ∗ nvinfer1::ICudaEngine::serialize ( ) const [inline], [noexcept]
Serialize the network to a stream.
Returns
A IHostMemory object that contains the serialized engine.
The network may be deserialized with IRuntime::deserializeCudaEngine().
See also
IRuntime::deserializeCudaEngine()
9.50.3.53 setErrorRecorder()
void nvinfer1::ICudaEngine::setErrorRecorder (
IErrorRecorder ∗ recorder ) [inline], [noexcept]
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function
will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder
with the interface, resulting in a call to decRefCount if a recorder has been registered.
If an error recorder is not set, messages will be sent to the global log stream.
Parameters
recorder
The error recorder to register with this interface.
See also
getErrorRecorder()
9.50.4
Member Data Documentation
244
Class Documentation
9.50.4.1
mImpl
apiv::VCudaEngine∗ nvinfer1::ICudaEngine::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInferRuntime.h
9.51
nvinfer1::safe::ICudaEngine Class Reference
A functionally safe engine for executing inference on a built network.
#include <NvInferSafeRuntime.h>
Public Member Functions
•
virtual TRT DEPRECATED std::int32 t getNbBindings () const noexcept=0
Get the number of binding indices.
•
virtual TRT DEPRECATED std::int32 t getBindingIndex (AsciiChar const ∗const name) const noexcept=0
Retrieve the binding index for a named tensor.
•
virtual TRT DEPRECATED AsciiChar const ∗ getBindingName (std::int32 t const bindingIndex) const noex-
cept=0
Retrieve the name corresponding to a binding index.
•
virtual TRT DEPRECATED bool bindingIsInput (std::int32 t const bindingIndex) const noexcept=0
Determine whether a binding is an input binding.
•
virtual TRT DEPRECATED Dims getBindingDimensions (std::int32 t const bindingIndex) const noexcept=0
Get the dimensions of a binding.
•
virtual TRT DEPRECATED DataType getBindingDataType (std::int32 t const bindingIndex) const noexcept=0
Determine the required data type for a buffer from its binding index.
•
virtual IExecutionContext ∗ createExecutionContext () noexcept=0
Create an execution context.
•
virtual IExecutionContext ∗ createExecutionContextWithoutDeviceMemory () noexcept=0
Create an execution context without any device memory allocated.
•
virtual size t getDeviceMemorySize () const noexcept=0
Return the amount of device memory required by an execution context.
•
virtual TRT DEPRECATED std::int32 t getBindingBytesPerComponent (std::int32 t const bindingIndex) const
noexcept=0
Return the number of bytes per component of an element.
•
virtual TRT DEPRECATED std::int32 t getBindingComponentsPerElement (std::int32 t const bindingIndex)
const noexcept=0
Return the number of components included in one element.
•
virtual TRT DEPRECATED TensorFormat getBindingFormat (std::int32 t const bindingIndex) const noex-
cept=0
Return the binding format.
•
virtual TRT DEPRECATED std::int32 t getBindingVectorizedDim (std::int32 t const bindingIndex) const noex-
cept=0
9.51 nvinfer1::safe::ICudaEngine Class Reference
245
Return the dimension index that the buffer is vectorized.
•
virtual AsciiChar const ∗ getName () const noexcept=0
Returns the name of the network associated with the engine.
•
virtual void setErrorRecorder (IErrorRecorder ∗const recorder) noexcept=0
Set the ErrorRecorder for this interface.
•
virtual IErrorRecorder ∗ getErrorRecorder () const noexcept=0
Get the ErrorRecorder assigned to this interface.
•
ICudaEngine ()=default
•
virtual ∼ICudaEngine () noexcept=default
•
ICudaEngine (ICudaEngine const &)=delete
•
ICudaEngine (ICudaEngine &&)=delete
•
ICudaEngine & operator= (ICudaEngine const &) &=delete
•
ICudaEngine & operator= (ICudaEngine &&) &=delete
•
virtual Dims getTensorShape (AsciiChar const ∗tensorName) const noexcept=0
Get extent of an input or output tensor.
•
virtual DataType getTensorDataType (AsciiChar const ∗tensorName) const noexcept=0
Determine the required data type for a buffer from its tensor name.
•
virtual TensorIOMode getTensorIOMode (AsciiChar const ∗tensorName) const noexcept=0
Determine whether a tensor is an input or output tensor.
•
virtual std::int32 t getTensorBytesPerComponent (AsciiChar const ∗tensorName) const noexcept=0
Return the number of bytes per component of an element.
•
virtual std::int32 t getTensorComponentsPerElement (AsciiChar const ∗tensorName) const noexcept=0
Return the number of components included in one element.
•
virtual TensorFormat getTensorFormat (AsciiChar const ∗tensorName) const noexcept=0
Return the tensor format.
•
virtual std::int32 t getTensorVectorizedDim (AsciiChar const ∗tensorName) const noexcept=0
Return the dimension index along which buffer is vectorized.
•
virtual std::int32 t getNbIOTensors () const noexcept=0
Return the number of input and output tensors for the network from which the engine was built.
•
virtual AsciiChar const ∗ getIOTensorName (std::int32 t const index) const noexcept=0
Return the name of an IO tensor.
9.51.1
Detailed Description
A functionally safe engine for executing inference on a built network.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.51.2
Constructor & Destructor Documentation
246
Class Documentation
9.51.2.1
ICudaEngine() [1/3]
nvinfer1::safe::ICudaEngine::ICudaEngine ( ) [default]
9.51.2.2
∼ICudaEngine()
virtual nvinfer1::safe::ICudaEngine::∼ICudaEngine ( ) [virtual], [default], [noexcept]
9.51.2.3
ICudaEngine() [2/3]
nvinfer1::safe::ICudaEngine::ICudaEngine (
ICudaEngine const &
) [delete]
9.51.2.4
ICudaEngine() [3/3]
nvinfer1::safe::ICudaEngine::ICudaEngine (
ICudaEngine &&
) [delete]
9.51.3
Member Function Documentation
9.51.3.1
bindingIsInput()
virtual TRT DEPRECATED bool nvinfer1::safe::ICudaEngine::bindingIsInput (
std::int32 t const bindingIndex ) const [pure virtual], [noexcept]
Determine whether a binding is an input binding.
Parameters
bindingIndex
The binding index.
Returns
True if the index corresponds to an input binding and the index is in range.
9.51 nvinfer1::safe::ICudaEngine Class Reference
247
Deprecated Deprecated in TensorRT 8.5. Superseded by tensorIOMode().
See also
safe::ICudaEngine::tensorIOMode()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.2
createExecutionContext()
virtual IExecutionContext ∗ nvinfer1::safe::ICudaEngine::createExecutionContext ( ) [pure virtual],
[noexcept]
Create an execution context.
See also
safe::IExecutionContext.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes; if createExecutionContext fails, users should treat this as a critical error and not perform
any subsequent TensorRT operations apart from outputting the error logs.
9.51.3.3
createExecutionContextWithoutDeviceMemory()
virtual IExecutionContext ∗ nvinfer1::safe::ICudaEngine::createExecutionContextWithoutDeviceMemory
( ) [pure virtual], [noexcept]
Create an execution context without any device memory allocated.
The memory for execution of this device context must be supplied by the application.
See also
getDeviceMemorySize() safe::IExecutionContext::setDeviceMemory()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes; if createExecutionContext fails, users should treat this as a critical error and not perform
any subsequent TensorRT operations apart from outputting the error logs.
248
Class Documentation
9.51.3.4
getBindingBytesPerComponent()
virtual TRT DEPRECATED std::int32 t nvinfer1::safe::ICudaEngine::getBindingBytesPerComponent (
std::int32 t const bindingIndex ) const [pure virtual], [noexcept]
Return the number of bytes per component of an element.
The vector component size is returned if getBindingVectorizedDim() != -1.
Parameters
bindingIndex
The binding Index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorBytesPerComponent().
See also
safe::ICudaEngine::getTensorBytesPerComponent()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.5
getBindingComponentsPerElement()
virtual TRT DEPRECATED std::int32 t nvinfer1::safe::ICudaEngine::getBindingComponentsPerElement (
std::int32 t const bindingIndex ) const [pure virtual], [noexcept]
Return the number of components included in one element.
The number of elements in the vectors is returned if getBindingVectorizedDim() != -1.
Parameters
bindingIndex
The binding Index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorComponentsPerElement().
9.51 nvinfer1::safe::ICudaEngine Class Reference
249
See also
safe::ICudaEngine::getTensorComponentsPerElement()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.6
getBindingDataType()
virtual TRT DEPRECATED DataType nvinfer1::safe::ICudaEngine::getBindingDataType (
std::int32 t const bindingIndex ) const [pure virtual], [noexcept]
Determine the required data type for a buffer from its binding index.
Parameters
bindingIndex
The binding index.
Returns
The type of the data in the buffer.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorDataType().
See also
safe::ICudaEngine::getTensorDataType()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.7
getBindingDimensions()
virtual TRT DEPRECATED Dims nvinfer1::safe::ICudaEngine::getBindingDimensions (
std::int32 t const bindingIndex ) const [pure virtual], [noexcept]
Get the dimensions of a binding.
250
Class Documentation
Parameters
bindingIndex
The binding index.
Returns
The dimensions of the binding if the index is in range, otherwise Dims()
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorShape().
See also
safe::ICudaEngine::getTensorShape()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.8
getBindingFormat()
virtual TRT DEPRECATED TensorFormat nvinfer1::safe::ICudaEngine::getBindingFormat (
std::int32 t const bindingIndex ) const [pure virtual], [noexcept]
Return the binding format.
Parameters
bindingIndex
The binding Index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorFormat().
See also
safe::ICudaEngine::getTensorFormat()
Usage considerations
9.51 nvinfer1::safe::ICudaEngine Class Reference
251
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.9
getBindingIndex()
virtual TRT DEPRECATED std::int32 t nvinfer1::safe::ICudaEngine::getBindingIndex (
AsciiChar const ∗const name ) const [pure virtual], [noexcept]
Retrieve the binding index for a named tensor.
safe::IExecutionContext::enqueueV2() requires an array of buffers. Engine bindings map from tensor names to indices
in this array. Binding indices are assigned at engine build time, and take values in the range [0 ... n-1] where n is the
total number of inputs and outputs.
Warning
Strings passed to the runtime must be 1024 characters or less including NULL terminator and must be
NULL terminated.
Parameters
name
The tensor name.
Returns
The binding index for the named tensor, or -1 if the name is not found.
Deprecated Deprecated in TensorRT 8.5. Superseded by name-based methods. Use them instead of binding-index
based methods.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.10 getBindingName()
virtual TRT DEPRECATED AsciiChar const ∗ nvinfer1::safe::ICudaEngine::getBindingName (
std::int32 t const bindingIndex ) const [pure virtual], [noexcept]
Retrieve the name corresponding to a binding index.
This is the reverse mapping to that provided by getBindingIndex().
252
Class Documentation
Parameters
bindingIndex
The binding index.
Returns
The name corresponding to the index, or nullptr if the index is out of range.
Deprecated Deprecated in TensorRT 8.5. Superseded by name-based methods. Use them instead of binding-index
based methods.
See also
getBindingIndex()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.11 getBindingVectorizedDim()
virtual TRT DEPRECATED std::int32 t nvinfer1::safe::ICudaEngine::getBindingVectorizedDim (
std::int32 t const bindingIndex ) const [pure virtual], [noexcept]
Return the dimension index that the buffer is vectorized.
Specifically -1 is returned if scalars per vector is 1.
Parameters
bindingIndex
The binding Index.
Deprecated Deprecated in TensorRT 8.5. Superseded by getTensorVectorizedDim().
See also
safe::ICudaEngine::getTensorVectorizedDim()
Usage considerations
9.51 nvinfer1::safe::ICudaEngine Class Reference
253
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.12 getDeviceMemorySize()
virtual size t nvinfer1::safe::ICudaEngine::getDeviceMemorySize ( ) const [pure virtual], [noexcept]
Return the amount of device memory required by an execution context.
See also
safe::IExecutionContext::setDeviceMemory()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.13 getErrorRecorder()
virtual IErrorRecorder ∗ nvinfer1::safe::ICudaEngine::getErrorRecorder ( ) const [pure virtual],
[noexcept]
Get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A nullptr will be returned if an error reporter has not
been inherited from the IRuntime, and setErrorReporter() has not been called.
Returns
A pointer to the IErrorRecorder object that has been registered.
See also
setErrorRecorder()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.14 getIOTensorName()
virtual AsciiChar const ∗ nvinfer1::safe::ICudaEngine::getIOTensorName (
std::int32 t const index ) const [pure virtual], [noexcept]
Return the name of an IO tensor.
If the index does not fall between 0 and getNbIOTensors()-1, the function will fail with an error code of
ErrorCode::kINVALID ARGUMENT(3) that is emitted to the registered IErrorRecorder.
254
Class Documentation
Parameters
index
The value that falls between 0 and getNbIOTensors()-1.
Returns
The name of an IO tensor. nullptr will be returned if the index does not fall between 0 and getNbIOTensors()-1.
See also
getNbIOTensors()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.15 getName()
virtual AsciiChar const ∗ nvinfer1::safe::ICudaEngine::getName ( ) const [pure virtual], [noexcept]
Returns the name of the network associated with the engine.
The name is set during network creation and is retrieved after building or deserialization.
See also
INetworkDefinition::setName(), INetworkDefinition::getName()
Returns
A null-terminated C-style string representing the name of the network.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51 nvinfer1::safe::ICudaEngine Class Reference
255
9.51.3.16 getNbBindings()
virtual TRT DEPRECATED std::int32 t nvinfer1::safe::ICudaEngine::getNbBindings ( ) const [pure
virtual], [noexcept]
Get the number of binding indices.
Returns
The number of binding indices.
Deprecated Deprecated in TensorRT 8.5. Superseded by getNbIOTensors.
See also
getBindingIndex()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.17 getNbIOTensors()
virtual std::int32 t nvinfer1::safe::ICudaEngine::getNbIOTensors ( ) const [pure virtual], [noexcept]
Return the number of input and output tensors for the network from which the engine was built.
Returns
The number of IO tensors.
See also
getIOTensorName()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.18 getTensorBytesPerComponent()
virtual std::int32 t nvinfer1::safe::ICudaEngine::getTensorBytesPerComponent (
AsciiChar const ∗ tensorName ) const [pure virtual], [noexcept]
Return the number of bytes per component of an element.
256
Class Documentation
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
Returns
The vector component size. 0 will be returned if (1) name is not the name of an input or output tensor, or (2)
name is nullptr, or (3) name exceeds the string length limit, or (4) the tensor of given name is not vectorized.
See also
safe::ICudaEngine::getTensorVectorizedDim()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.51.3.19 getTensorComponentsPerElement()
virtual std::int32 t nvinfer1::safe::ICudaEngine::getTensorComponentsPerElement (
AsciiChar const ∗ tensorName ) const [pure virtual], [noexcept]
Return the number of components included in one element.
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
|
||
|
|
|