|
|
482
Class Documentation
Warning
The string inputName must be null-terminated, and be at most 4096 bytes including the terminator.
9.95.3.2
getExtraMemoryTarget()
float nvinfer1::IOptimizationProfile::getExtraMemoryTarget ( ) const [inline], [noexcept]
Get the extra memory target that has been defined for this profile.
This defaults to 1.0F.
Returns
the valid value set by setExtraMemoryTarget or 1.0F.
9.95.3.3
getNbShapeValues()
int32 t nvinfer1::IOptimizationProfile::getNbShapeValues (
char const ∗ inputName ) const [inline], [noexcept]
Get the number of values for an input shape tensor.
This will return the number of shape values if setShapeValues() has been called before for this input tensor. Otherwise,
return -1.
Warning
The string inputName must be null-terminated, and be at most 4096 bytes including the terminator.
9.95.3.4
getShapeValues()
int32 t const ∗ nvinfer1::IOptimizationProfile::getShapeValues (
char const ∗ inputName,
OptProfileSelector select ) const [inline], [noexcept]
Get the minimum / optimum / maximum values for an input shape tensor.
If the shape values have not been set previously with setShapeValues(), this returns nullptr.
9.95 nvinfer1::IOptimizationProfile Class Reference
483
Warning
The string inputName must be null-terminated, and be at most 4096 bytes including the terminator.
9.95.3.5
isValid()
bool nvinfer1::IOptimizationProfile::isValid ( ) const [inline], [noexcept]
Check whether the optimization profile can be passed to an IBuilderConfig object.
This function performs partial validation, by e.g. checking that whenever one of the minimum, optimum, or maximum
dimensions of a tensor have been set, the others have also been set and have the same rank, as well as checking that the
optimum dimensions are always as least as large as the minimum dimensions, and that the maximum dimensions are
at least as large as the optimum dimensions. Some validation steps require knowledge of the network definition and
are deferred to engine build time.
Returns
true if the optimization profile is valid and may be passed to an IBuilderConfig, else false.
9.95.3.6
setDimensions()
bool nvinfer1::IOptimizationProfile::setDimensions (
char const ∗ inputName,
OptProfileSelector select,
Dims dims ) [inline], [noexcept]
Set the minimum / optimum / maximum dimensions for a dynamic input tensor.
This function must be called three times (for the minimum, optimum, and maximum) for any network input tensor that
has dynamic dimensions. If minDims, optDims, and maxDims are the minimum, optimum, and maximum dimensions,
and networkDims are the dimensions for this input tensor that are provided to the INetworkDefinition object, then the
following conditions must all hold:
(1) minDims.nbDims == optDims.nbDims == maxDims.nbDims == networkDims.nbDims (2) 0 <= minDims.d[i] <=
optDims.d[i] <= maxDims.d[i] for i = 0, ..., networkDims.nbDims-1 (3) if networkDims.d[i] != -1, then minDims.d[i]
== optDims.d[i] == maxDims.d[i] == networkDims.d[i]
This function may (but need not be) called for an input tensor that does not have dynamic dimensions. In this case, the
third argument must always equal networkDims.
Parameters
inputName
The input tensor name
select
Whether to set the minimum, optimum, or maximum dimensions
dims
The minimum, optimum, or maximum dimensions for this input tensor
484
Class Documentation
Returns
false if an inconsistency was detected (e.g. the rank does not match another dimension that was previously set
for the same input), true if no inconsistency was detected. Note that inputs can be validated only partially; a full
validation is performed at engine build time.
Warning
If run on DLA, minimum, optimum, and maximum dimensions must to be the same.
The string inputName must be null-terminated, and be at most 4096 bytes including the terminator.
9.95.3.7
setExtraMemoryTarget()
bool nvinfer1::IOptimizationProfile::setExtraMemoryTarget (
float target ) [inline], [noexcept]
Set a target for extra GPU memory that may be used by this profile.
Parameters
target
Additional memory that the builder should aim to maximally allocate for this profile, as a fraction of the
memory it would use if the user did not impose any constraints on memory. This unconstrained case is the
default; it corresponds to target == 1.0. If target == 0.0, the builder aims to create the new optimization
profile without allocating any additional weight memory. Valid inputs lie between 0.0 and 1.0. This
parameter is only a hint, and TensorRT does not guarantee that the target will be reached. This parameter
is ignored for the first (default) optimization profile that is defined.
Returns
true if the input is in the valid range (between 0 and 1 inclusive), else false.
9.95.3.8
setShapeValues()
bool nvinfer1::IOptimizationProfile::setShapeValues (
char const ∗ inputName,
OptProfileSelector select,
int32 t const ∗ values,
int32 t nbValues ) [inline], [noexcept]
Set the minimum / optimum / maximum values for an input shape tensor.
This function must be called three times for every input tensor t that is a shape tensor (t.isShape() == true). This
implies that the datatype of t is DataType::kINT32, the rank is either 0 or 1, and the dimensions of t are fixed at
network definition time. This function must not be called for any input tensor that is not a shape tensor.
9.95 nvinfer1::IOptimizationProfile Class Reference
485
Each time this function is called for the same input tensor, the same nbValues must be supplied (either 1 if the tensor
rank is 0, or dims.d[0] if the rank is 1). Furthermore, if minVals, optVals, maxVals are the minimum, optimum, and
maximum values, it must be true that minVals[i] <= optVals[i] <= maxVals[i] for i = 0, ..., nbValues - 1. Execution of
the network must be valid for the optVals.
Shape tensors are tensors that contribute to shape calculations in some way, and can contain any int32 t values appro-
priate for the network. Shape tensors of other data types (e.g. float) are not supported. Examples:
• A shape tensor used as the second input to IShuffleLayer can contain a -1 wildcard. The corresponding minVal[i]
should be -1.
• A shape tensor used as the stride input to ISliceLayer can contain any valid strides. The values could be positive,
negative, or zero.
• A shape tensor subtracted from zero to compute the size input of an ISliceLayer can contain any non-positive
values that yield a valid slice operation.
Tightening the minVals and maxVals bounds to cover only values that are necessary may help optimization.
Parameters
inputName
The input tensor name
select
Whether to set the minimum, optimum, or maximum input values.
values
An array of length nbValues containing the minimum, optimum, or maximum shape tensor elements.
nbValues
The length of the value array, which must equal the number of shape tensor elements (>= 1)
Returns
false if an inconsistency was detected (e.g. nbValues does not match a previous call for the same tensor), else
true. As for setDimensions(), a full validation can only be performed at engine build time.
Warning
If run on DLA, minimum, optimum, and maximum shape values must to be the same.
The string inputName must be null-terminated, and be at most 4096 bytes including the terminator.
9.95.4
Member Data Documentation
9.95.4.1
mImpl
apiv::VOptimizationProfile∗ nvinfer1::IOptimizationProfile::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInferRuntime.h
486
Class Documentation
9.96
nvinfer1::IOutputAllocator Class Reference
Callback from ExecutionContext::enqueueV3()
#include <NvInferRuntime.h>
Public Member Functions
• virtual int32 t getInterfaceVersion () const noexcept
Return the API version of this IOutputAllocator.
• virtual void ∗ reallocateOutput (char const ∗tensorName, void ∗currentMemory, uint64 t size, uint64 t align-
ment) noexcept=0
Return a pointer to memory for an output tensor, or nullptr if memory cannot be allocated.
• virtual void notifyShape (char const ∗tensorName, Dims const &dims) noexcept=0
Called by TensorRT when the shape of the output tensor is known.
• virtual ∼IOutputAllocator ()=default
9.96.1
Detailed Description
Callback from ExecutionContext::enqueueV3()
Clients should override the method reallocateOutput.
See also
IExecutionContext::enqueueV3()
9.96.2
Constructor & Destructor Documentation
9.96.2.1
∼IOutputAllocator()
virtual nvinfer1::IOutputAllocator::∼IOutputAllocator ( ) [virtual], [default]
9.96.3
Member Function Documentation
9.96 nvinfer1::IOutputAllocator Class Reference
487
9.96.3.1
getInterfaceVersion()
virtual int32 t nvinfer1::IOutputAllocator::getInterfaceVersion ( ) const [inline], [virtual],
[noexcept]
Return the API version of this IOutputAllocator.
Do not override this method as it is used by the TensorRT library to maintain backwards-compatibility with
IOutputAllocator. The value will change if Nvidia adds additional virtual methods to this class.
9.96.3.2
notifyShape()
virtual void nvinfer1::IOutputAllocator::notifyShape (
char const ∗ tensorName,
Dims const & dims ) [pure virtual], [noexcept]
Called by TensorRT when the shape of the output tensor is known.
Called by TensorRT sometime between when it calls reallocateOutput and enqueueV3 returns.
Parameters
dims
dimensions of the output
tensorName
name of the tensor
9.96.3.3
reallocateOutput()
virtual void ∗ nvinfer1::IOutputAllocator::reallocateOutput (
char const ∗ tensorName,
void ∗ currentMemory,
uint64 t size,
uint64 t alignment ) [pure virtual], [noexcept]
Return a pointer to memory for an output tensor, or nullptr if memory cannot be allocated.
Parameters
tensorName
name of the output tensor.
currentMemory
points to the address set by IExectionContext::setTensorAddress.
size
number of bytes required. Always positive, even for an empty tensor.
alignment
required alignment of the allocation.
488
Class Documentation
Returns
A pointer to memory to use for the output tensor or nullptr.
If currentMemory is known to be big enough, one option is to return currentMemory.
To preallocate memory and have the engine fail if the preallocation is not big enough, use IExecutionContext::setTensorAddress
to set a pointer to the preallocated memory, and have reallocateOutput return nullptr if that memory is not big enough.
The documentation for this class was generated from the following file:
• NvInferRuntime.h
9.97
nvinfer1::IPaddingLayer Class Reference
Layer that represents a padding operation.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IPaddingLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IPaddingLayer
Public Member Functions
• TRT DEPRECATED void setPrePadding (DimsHW padding) noexcept
Set the padding that is applied at the start of the tensor.
• TRT DEPRECATED DimsHW getPrePadding () const noexcept
Get the padding that is applied at the start of the tensor.
• TRT DEPRECATED void setPostPadding (DimsHW padding) noexcept
Set the padding that is applied at the end of the tensor.
• TRT DEPRECATED DimsHW getPostPadding () const noexcept
Get the padding that is applied at the end of the tensor.
• void setPrePaddingNd (Dims padding) noexcept
Set the padding that is applied at the start of the tensor.
• Dims getPrePaddingNd () const noexcept
Get the padding that is applied at the start of the tensor.
• void setPostPaddingNd (Dims padding) noexcept
Set the padding that is applied at the end of the tensor.
• Dims getPostPaddingNd () const noexcept
Get the padding that is applied at the end of the tensor.
9.97 nvinfer1::IPaddingLayer Class Reference
489
Protected Member Functions
• virtual ∼IPaddingLayer () noexcept=default
Protected Attributes
• apiv::VPaddingLayer ∗ mImpl
9.97.1
Detailed Description
Layer that represents a padding operation.
The padding layer adds zero-padding at the start and end of the input tensor. It only supports padding along the two
innermost dimensions. Applying negative padding results in cropping of the input.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.97.2
Constructor & Destructor Documentation
9.97.2.1
∼IPaddingLayer()
virtual nvinfer1::IPaddingLayer::∼IPaddingLayer ( ) [protected], [virtual], [default], [noexcept]
9.97.3
Member Function Documentation
9.97.3.1
getPostPadding()
TRT DEPRECATED DimsHW nvinfer1::IPaddingLayer::getPostPadding ( ) const [inline], [noexcept]
Get the padding that is applied at the end of the tensor.
See also
setPostPadding
Deprecated Superseded by getPostPaddingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.97.3.2
getPostPaddingNd()
Dims nvinfer1::IPaddingLayer::getPostPaddingNd ( ) const [inline], [noexcept]
Get the padding that is applied at the end of the tensor.
490
Class Documentation
Warning
Only 2 dimensional padding is currently supported.
See also
setPostPaddingNd
9.97.3.3
getPrePadding()
TRT DEPRECATED DimsHW nvinfer1::IPaddingLayer::getPrePadding ( ) const [inline], [noexcept]
Get the padding that is applied at the start of the tensor.
See also
setPrePadding
Deprecated Superseded by getPrePaddingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.97.3.4
getPrePaddingNd()
Dims nvinfer1::IPaddingLayer::getPrePaddingNd ( ) const [inline], [noexcept]
Get the padding that is applied at the start of the tensor.
Warning
Only 2 dimensional padding is currently supported.
See also
setPrePaddingNd
9.97 nvinfer1::IPaddingLayer Class Reference
491
9.97.3.5
setPostPadding()
TRT DEPRECATED void nvinfer1::IPaddingLayer::setPostPadding (
DimsHW padding ) [inline], [noexcept]
Set the padding that is applied at the end of the tensor.
Negative padding results in trimming the edge by the specified amount
See also
getPostPadding
Deprecated Superseded by setPostPaddingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.97.3.6
setPostPaddingNd()
void nvinfer1::IPaddingLayer::setPostPaddingNd (
Dims padding ) [inline], [noexcept]
Set the padding that is applied at the end of the tensor.
Negative padding results in trimming the edge by the specified amount
Warning
Only 2 dimensional padding is currently supported.
See also
getPostPaddingNd
9.97.3.7
setPrePadding()
TRT DEPRECATED void nvinfer1::IPaddingLayer::setPrePadding (
DimsHW padding ) [inline], [noexcept]
Set the padding that is applied at the start of the tensor.
Negative padding results in trimming the edge by the specified amount
See also
getPrePadding
Deprecated Superseded by setPrePaddingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
492
Class Documentation
9.97.3.8
setPrePaddingNd()
void nvinfer1::IPaddingLayer::setPrePaddingNd (
Dims padding ) [inline], [noexcept]
Set the padding that is applied at the start of the tensor.
Negative padding results in trimming the edge by the specified amount.
Warning
Only 2 dimensional padding is currently supported.
See also
getPrePaddingNd
9.97.4
Member Data Documentation
9.97.4.1
mImpl
apiv::VPaddingLayer∗ nvinfer1::IPaddingLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.98
nvinfer1::IParametricReLULayer Class Reference
Layer that represents a parametric ReLU operation.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IParametricReLULayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IParametricReLULayer
9.98 nvinfer1::IParametricReLULayer Class Reference
493
Protected Member Functions
• virtual ∼IParametricReLULayer () noexcept=default
Protected Attributes
• apiv::VParametricReLULayer ∗ mImpl
Additional Inherited Members
9.98.1
Detailed Description
Layer that represents a parametric ReLU operation.
When running this layer on DLA, the slopes input must be a build-time constant.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.98.2
Constructor & Destructor Documentation
9.98.2.1
∼IParametricReLULayer()
virtual nvinfer1::IParametricReLULayer::∼IParametricReLULayer ( ) [protected], [virtual], [default],
[noexcept]
9.98.3
Member Data Documentation
9.98.3.1
mImpl
apiv::VParametricReLULayer∗ nvinfer1::IParametricReLULayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
494
Class Documentation
9.99
nvonnxparser::IParser Class Reference
an object for parsing ONNX models into a TensorRT network definition
#include <NvOnnxParser.h>
Public Member Functions
•
virtual bool parse (void const ∗serialized onnx model, size t serialized onnx model size, const char ∗model ←↩
path=nullptr)=0
Parse a serialized ONNX model into the TensorRT network. This method has very limited diagnostics. If parsing the
serialized model fails for any reason (e.g. unsupported IR version, unsupported opset, etc.) it the user responsibility to
intercept and report the error. To obtain a better diagnostic, use the parseFromFile method below.
•
virtual bool parseFromFile (const char ∗onnxModelFile, int verbosity)=0
Parse an onnx model file, which can be a binary protobuf or a text onnx model calls parse method inside.
•
virtual bool supportsModel (void const ∗serialized onnx model, size t serialized onnx model size, SubGraphCollection t
&sub graph collection, const char ∗model path=nullptr)=0
Check whether TensorRT supports a particular ONNX model. If the function returns True, one can proceed to engine
building without having to call parse or parseFromFile.
•
virtual bool parseWithWeightDescriptors (void const ∗serialized onnx model, size t serialized onnx model ←↩
size)=0
Parse a serialized ONNX model into the TensorRT network with consideration of user provided weights.
•
virtual bool supportsOperator (const char ∗op name) const =0
Returns whether the specified operator may be supported by the parser.
•
virtual TRT DEPRECATED void destroy ()=0
destroy this object
•
virtual int getNbErrors () const =0
Get the number of errors that occurred during prior calls to parse.
•
virtual IParserError const ∗ getError (int index) const =0
Get an error that occurred during prior calls to parse.
•
virtual void clearErrors ()=0
Clear errors from prior calls to parse.
•
virtual ∼IParser () noexcept=default
•
virtual char const ∗const ∗ getUsedVCPluginLibraries (int64 t &nbPluginLibs) const noexcept=0
Query the plugin libraries needed to implement operations used by the parser in a version-compatible engine.
•
virtual void setFlags (OnnxParserFlags onnxParserFlags) noexcept=0
Set the parser flags.
•
virtual OnnxParserFlags getFlags () const noexcept=0
Get the parser flags. Defaults to 0.
•
virtual void clearFlag (OnnxParserFlag onnxParserFlag) noexcept=0
clear a parser flag.
•
virtual void setFlag (OnnxParserFlag onnxParserFlag) noexcept=0
Set a single parser flag.
•
virtual bool getFlag (OnnxParserFlag onnxParserFlag) const noexcept=0
Returns true if the parser flag is set.
9.99 nvonnxparser::IParser Class Reference
495
9.99.1
Detailed Description
an object for parsing ONNX models into a TensorRT network definition
9.99.2
Constructor & Destructor Documentation
9.99.2.1
∼IParser()
virtual nvonnxparser::IParser::∼IParser ( ) [virtual], [default], [noexcept]
9.99.3
Member Function Documentation
9.99.3.1
clearErrors()
virtual void nvonnxparser::IParser::clearErrors ( ) [pure virtual]
Clear errors from prior calls to parse.
See also
getNbErrors() getError() IParserError
9.99.3.2
clearFlag()
virtual void nvonnxparser::IParser::clearFlag (
OnnxParserFlag onnxParserFlag ) [pure virtual], [noexcept]
clear a parser flag.
clears the parser flag from the enabled flags.
See also
setFlags()
9.99.3.3
destroy()
virtual TRT DEPRECATED void nvonnxparser::IParser::destroy ( ) [pure virtual]
destroy this object
496
Class Documentation
Warning
deprecated and planned on being removed in TensorRT 10.0
9.99.3.4
getError()
virtual IParserError const ∗ nvonnxparser::IParser::getError (
int index ) const [pure virtual]
Get an error that occurred during prior calls to parse.
See also
getNbErrors() clearErrors() IParserError
9.99.3.5
getFlag()
virtual bool nvonnxparser::IParser::getFlag (
OnnxParserFlag onnxParserFlag ) const [pure virtual], [noexcept]
Returns true if the parser flag is set.
See also
getFlags()
Returns
True if flag is set, false if unset.
9.99.3.6
getFlags()
virtual OnnxParserFlags nvonnxparser::IParser::getFlags ( ) const [pure virtual], [noexcept]
Get the parser flags. Defaults to 0.
Returns
The parser flags as a bitmask.
See also
setFlags()
9.99 nvonnxparser::IParser Class Reference
497
9.99.3.7
getNbErrors()
virtual int nvonnxparser::IParser::getNbErrors ( ) const [pure virtual]
Get the number of errors that occurred during prior calls to parse.
See also
getError() clearErrors() IParserError
9.99.3.8
getUsedVCPluginLibraries()
virtual char const ∗const ∗ nvonnxparser::IParser::getUsedVCPluginLibraries (
int64 t & nbPluginLibs ) const [pure virtual], [noexcept]
Query the plugin libraries needed to implement operations used by the parser in a version-compatible engine.
This provides a list of plugin libraries on the filesystem needed to implement operations in the parsed network. If you
are building a version-compatible engine using this network, provide this list to IBuilderConfig::setPluginsToSerialize
to serialize these plugins along with the version-compatible engine, or, if you want to ship these plugin libraries
externally to the engine, ensure that IPluginRegistry::loadLibrary is used to load these libraries in the appropriate
runtime before deserializing the corresponding engine.
Parameters
out
nbPluginLibs
Returns the number of plugin libraries in the array, or -1 if there was an error.
Returns
Array of nbPluginLibs C-strings describing plugin library paths on the filesystem if nbPluginLibs > 0, or
nullptr otherwise. This array is owned by the IParser, and the pointers in the array are only valid until the next
call to parse(), supportsModel(), parseFromFile(), or parseWithWeightDescriptors().
9.99.3.9
parse()
virtual bool nvonnxparser::IParser::parse (
void const ∗ serialized onnx model,
size t serialized onnx model size,
const char ∗ model path = nullptr ) [pure virtual]
Parse a serialized ONNX model into the TensorRT network. This method has very limited diagnostics. If parsing the
serialized model fails for any reason (e.g. unsupported IR version, unsupported opset, etc.) it the user responsibility to
intercept and report the error. To obtain a better diagnostic, use the parseFromFile method below.
498
Class Documentation
Parameters
serialized onnx model
Pointer to the serialized ONNX model
serialized onnx model size
Size of the serialized ONNX model in bytes
model path
Absolute path to the model file for loading external weights if required
Returns
true if the model was parsed successfully
See also
getNbErrors() getError()
9.99.3.10 parseFromFile()
virtual bool nvonnxparser::IParser::parseFromFile (
const char ∗ onnxModelFile,
int verbosity ) [pure virtual]
Parse an onnx model file, which can be a binary protobuf or a text onnx model calls parse method inside.
Parameters
onnxModelFile
name
verbosity
Level
Returns
true if the model was parsed successfully
9.99.3.11 parseWithWeightDescriptors()
virtual bool nvonnxparser::IParser::parseWithWeightDescriptors (
void const ∗ serialized onnx model,
size t serialized onnx model size ) [pure virtual]
Parse a serialized ONNX model into the TensorRT network with consideration of user provided weights.
9.99 nvonnxparser::IParser Class Reference
499
Parameters
serialized onnx model
Pointer to the serialized ONNX model
serialized onnx model size
Size of the serialized ONNX model in bytes
Returns
true if the model was parsed successfully
See also
getNbErrors() getError()
9.99.3.12 setFlag()
virtual void nvonnxparser::IParser::setFlag (
OnnxParserFlag onnxParserFlag ) [pure virtual], [noexcept]
Set a single parser flag.
Add the input parser flag to the already enabled flags.
See also
setFlags()
9.99.3.13 setFlags()
virtual void nvonnxparser::IParser::setFlags (
OnnxParserFlags onnxParserFlags ) [pure virtual], [noexcept]
Set the parser flags.
The flags are listed in the OnnxParserFlag enum.
Parameters
OnnxParserFlag
The flags used when parsing an ONNX model.
500
Class Documentation
Note
This function will override the previous set flags, rather than bitwise ORing the new flag.
See also
getFlags()
9.99.3.14 supportsModel()
virtual bool nvonnxparser::IParser::supportsModel (
void const ∗ serialized onnx model,
size t serialized onnx model size,
SubGraphCollection t & sub graph collection,
const char ∗ model path = nullptr ) [pure virtual]
Check whether TensorRT supports a particular ONNX model. If the function returns True, one can proceed to engine
building without having to call parse or parseFromFile.
Parameters
serialized onnx model
Pointer to the serialized ONNX model
serialized onnx model size
Size of the serialized ONNX model in bytes
sub graph collection
Container to hold supported subgraphs
model path
Absolute path to the model file for loading external weights if required
Returns
true if the model is supported
9.99.3.15 supportsOperator()
virtual bool nvonnxparser::IParser::supportsOperator (
const char ∗ op name ) const [pure virtual]
Returns whether the specified operator may be supported by the parser.
Note that a result of true does not guarantee that the operator will be supported in all cases (i.e., this function may
return false-positives).
Parameters
op name
The name of the ONNX operator to check for support
9.100 nvonnxparser::IParserError Class Reference
501
The documentation for this class was generated from the following file:
• NvOnnxParser.h
9.100
nvonnxparser::IParserError Class Reference
an object containing information about an error
#include <NvOnnxParser.h>
Public Member Functions
• virtual ErrorCode code () const =0
the error code
• virtual const char ∗ desc () const =0
description of the error
• virtual const char ∗ file () const =0
source file in which the error occurred
• virtual int line () const =0
source line at which the error occurred
• virtual const char ∗ func () const =0
source function in which the error occurred
• virtual int node () const =0
index of the ONNX model node in which the error occurred
Protected Member Functions
• virtual ∼IParserError ()
9.100.1
Detailed Description
an object containing information about an error
9.100.2
Constructor & Destructor Documentation
9.100.2.1 ∼IParserError()
virtual nvonnxparser::IParserError::∼IParserError ( ) [inline], [protected], [virtual]
502
Class Documentation
9.100.3
Member Function Documentation
9.100.3.1 code()
virtual ErrorCode nvonnxparser::IParserError::code ( ) const [pure virtual]
the error code
9.100.3.2 desc()
virtual const char ∗ nvonnxparser::IParserError::desc ( ) const [pure virtual]
description of the error
9.100.3.3 file()
virtual const char ∗ nvonnxparser::IParserError::file ( ) const [pure virtual]
source file in which the error occurred
9.100.3.4 func()
virtual const char ∗ nvonnxparser::IParserError::func ( ) const [pure virtual]
source function in which the error occurred
9.100.3.5 line()
virtual int nvonnxparser::IParserError::line ( ) const [pure virtual]
source line at which the error occurred
9.101 nvinfer1::consistency::IPluginChecker Class Reference
503
9.100.3.6 node()
virtual int nvonnxparser::IParserError::node ( ) const [pure virtual]
index of the ONNX model node in which the error occurred
The documentation for this class was generated from the following file:
• NvOnnxParser.h
9.101
nvinfer1::consistency::IPluginChecker Class Reference
Consistency Checker plugin class for user implemented Plugins.
#include <NvInferConsistency.h>
Inheritance diagram for nvinfer1::consistency::IPluginChecker:
nvinfer1::IPluginCreator
nvinfer1::consistency::IPluginChecker
Public Member Functions
• virtual bool validate (char const ∗name, void const ∗serialData, size t serialLength, PluginTensorDesc const ∗in,
size t nbInputs, PluginTensorDesc const ∗out, size t nbOutputs, int64 t workspaceSize) const noexcept=0
Called during IConsistencyChecker::validate. Allows users to provide custom validation of serialized Plugin data. Re-
turns boolean that indicates whether or not the Plugin passed validation.
• IPluginChecker ()=default
• virtual ∼IPluginChecker () override=default
Protected Member Functions
• IPluginChecker (IPluginChecker const &)=default
• IPluginChecker (IPluginChecker &&)=default
• IPluginChecker & operator= (IPluginChecker const &) &=default
• IPluginChecker & operator= (IPluginChecker &&) &=default
9.101.1
Detailed Description
Consistency Checker plugin class for user implemented Plugins.
Plugins are a mechanism for applications to implement custom layers. It provides a mechanism to register Consistency
plugins and look up the Plugin Registry during validate.
Supported IPlugin inferfaces are limited to IPluginV2IOExt only.
504
Class Documentation
9.101.2
Constructor & Destructor Documentation
9.101.2.1 IPluginChecker() [1/3]
nvinfer1::consistency::IPluginChecker::IPluginChecker ( ) [default]
9.101.2.2 ∼IPluginChecker()
virtual nvinfer1::consistency::IPluginChecker::∼IPluginChecker ( ) [override], [virtual], [default]
9.101.2.3 IPluginChecker() [2/3]
nvinfer1::consistency::IPluginChecker::IPluginChecker (
IPluginChecker const &
) [protected], [default]
9.101.2.4 IPluginChecker() [3/3]
nvinfer1::consistency::IPluginChecker::IPluginChecker (
IPluginChecker &&
) [protected], [default]
9.101.3
Member Function Documentation
9.101.3.1 operator=() [1/2]
IPluginChecker & nvinfer1::consistency::IPluginChecker::operator= (
IPluginChecker &&
) & [protected], [default]
9.102 nvinfer1::IPluginCreator Class Reference
505
9.101.3.2 operator=() [2/2]
IPluginChecker & nvinfer1::consistency::IPluginChecker::operator= (
IPluginChecker const &
) & [protected], [default]
9.101.3.3 validate()
virtual bool nvinfer1::consistency::IPluginChecker::validate (
char const ∗ name,
void const ∗ serialData,
size t serialLength,
PluginTensorDesc const ∗ in,
size t nbInputs,
PluginTensorDesc const ∗ out,
size t nbOutputs,
int64 t workspaceSize ) const [pure virtual], [noexcept]
Called during IConsistencyChecker::validate. Allows users to provide custom validation of serialized Plugin data.
Returns boolean that indicates whether or not the Plugin passed validation.
Parameters
name
The plugin name
serialData
The memory that holds the plugin serialized data.
serialLength
The size of the plugin serialized data.
in
The input tensors attributes.
nbInputs
The number of input tensors.
out
The output tensors attributes.
nbOutputs
The number of output tensors.
workspaceSize
The size of workspace provided during enqueue.
The documentation for this class was generated from the following file:
• NvInferConsistency.h
9.102
nvinfer1::IPluginCreator Class Reference
Plugin creator class for user implemented layers.
#include <NvInferRuntimePlugin.h>
Inheritance diagram for nvinfer1::IPluginCreator:
506
Class Documentation
nvinfer1::IPluginCreator
nvinfer1::consistency::IPluginChecker
Public Member Functions
• virtual int32 t getTensorRTVersion () const noexcept
Return the version of the API the plugin creator was compiled with.
• virtual AsciiChar const ∗ getPluginName () const noexcept=0
Return the plugin name.
• virtual AsciiChar const ∗ getPluginVersion () const noexcept=0
Return the plugin version.
• virtual PluginFieldCollection const ∗ getFieldNames () noexcept=0
Return a list of fields that needs to be passed to createPlugin.
• virtual IPluginV2 ∗ createPlugin (AsciiChar const ∗name, PluginFieldCollection const ∗fc) noexcept=0
Return a plugin object. Return nullptr in case of error.
• virtual IPluginV2 ∗ deserializePlugin (AsciiChar const ∗name, void const ∗serialData, size t serialLength) noex-
cept=0
Called during deserialization of plugin layer. Return a plugin object.
• virtual void setPluginNamespace (AsciiChar const ∗pluginNamespace) noexcept=0
Set the namespace of the plugin creator based on the plugin library it belongs to. This can be set while registering the
plugin creator.
• virtual AsciiChar const ∗ getPluginNamespace () const noexcept=0
Return the namespace of the plugin creator object.
• IPluginCreator ()=default
• virtual ∼IPluginCreator ()=default
9.102.1
Detailed Description
Plugin creator class for user implemented layers.
See also
IPlugin and IPluginFactory
9.102.2
Constructor & Destructor Documentation
9.102.2.1 IPluginCreator()
nvinfer1::IPluginCreator::IPluginCreator ( ) [default]
9.102 nvinfer1::IPluginCreator Class Reference
507
9.102.2.2 ∼IPluginCreator()
virtual nvinfer1::IPluginCreator::∼IPluginCreator ( ) [virtual], [default]
9.102.3
Member Function Documentation
9.102.3.1 createPlugin()
virtual IPluginV2 ∗ nvinfer1::IPluginCreator::createPlugin (
AsciiChar const ∗ name,
PluginFieldCollection const ∗ fc ) [pure virtual], [noexcept]
Return a plugin object. Return nullptr in case of error.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin or when deserializing multiple engines
concurrently sharing plugins.
9.102.3.2 deserializePlugin()
virtual IPluginV2 ∗ nvinfer1::IPluginCreator::deserializePlugin (
AsciiChar const ∗ name,
void const ∗ serialData,
size t serialLength ) [pure virtual], [noexcept]
Called during deserialization of plugin layer. Return a plugin object.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin or when deserializing multiple engines
concurrently sharing plugins.
508
Class Documentation
9.102.3.3 getFieldNames()
virtual PluginFieldCollection const ∗ nvinfer1::IPluginCreator::getFieldNames ( ) [pure virtual],
[noexcept]
Return a list of fields that needs to be passed to createPlugin.
See also
PluginFieldCollection
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin or when deserializing multiple engines
concurrently sharing plugins.
9.102.3.4 getPluginName()
virtual AsciiChar const ∗ nvinfer1::IPluginCreator::getPluginName ( ) const [pure virtual], [noexcept]
Return the plugin name.
Warning
The string returned must be 1024 bytes or less including the NULL terminator and must be NULL termi-
nated.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin or when deserializing multiple engines
concurrently sharing plugins.
9.102.3.5 getPluginNamespace()
virtual AsciiChar const ∗ nvinfer1::IPluginCreator::getPluginNamespace ( ) const [pure virtual],
[noexcept]
Return the namespace of the plugin creator object.
9.102 nvinfer1::IPluginCreator Class Reference
509
Warning
The string returned must be 1024 bytes or less including the NULL terminator and must be NULL termi-
nated.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin or when deserializing multiple engines
concurrently sharing plugins.
9.102.3.6 getPluginVersion()
virtual AsciiChar const ∗ nvinfer1::IPluginCreator::getPluginVersion ( ) const [pure virtual],
[noexcept]
Return the plugin version.
Warning
The string returned must be 1024 bytes or less including the NULL terminator and must be NULL termi-
nated.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin or when deserializing multiple engines
concurrently sharing plugins.
9.102.3.7 getTensorRTVersion()
virtual int32 t nvinfer1::IPluginCreator::getTensorRTVersion ( ) const [inline], [virtual], [noexcept]
Return the version of the API the plugin creator was compiled with.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, the implementation provided here is safe to call from any thread.
510
Class Documentation
9.102.3.8 setPluginNamespace()
virtual void nvinfer1::IPluginCreator::setPluginNamespace (
AsciiChar const ∗ pluginNamespace ) [pure virtual], [noexcept]
Set the namespace of the plugin creator based on the plugin library it belongs to. This can be set while registering the
plugin creator.
See also
IPluginRegistry::registerCreator()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin or when deserializing multiple engines
concurrently sharing plugins.
The documentation for this class was generated from the following file:
• NvInferRuntimePlugin.h
9.103
nvcaffeparser1::IPluginFactoryV2 Class Reference
Plugin factory used to configure plugins.
#include <NvCaffeParser.h>
Public Member Functions
• virtual bool isPluginV2 (char const ∗layerName) noexcept=0
A user implemented function that determines if a layer configuration is provided by an IPluginV2.
• virtual nvinfer1::IPluginV2 ∗ createPlugin (char const ∗layerName, nvinfer1::Weights const ∗weights, int32 t
nbWeights, char const ∗libNamespace=””) noexcept=0
Creates a plugin.
• virtual ∼IPluginFactoryV2 () noexcept=default
9.103.1
Detailed Description
Plugin factory used to configure plugins.
9.103 nvcaffeparser1::IPluginFactoryV2 Class Reference
511
9.103.2
Constructor & Destructor Documentation
9.103.2.1 ∼IPluginFactoryV2()
virtual nvcaffeparser1::IPluginFactoryV2::∼IPluginFactoryV2 ( ) [virtual], [default], [noexcept]
9.103.3
Member Function Documentation
9.103.3.1 createPlugin()
virtual nvinfer1::IPluginV2 ∗ nvcaffeparser1::IPluginFactoryV2::createPlugin (
char const ∗ layerName,
nvinfer1::Weights const ∗ weights,
int32 t nbWeights,
char const ∗ libNamespace = "" ) [pure virtual], [noexcept]
Creates a plugin.
Parameters
layerName
Name of layer associated with the plugin.
weights
Weights used for the layer.
nbWeights
Number of weights.
libNamespace
Library Namespace associated with the plugin object
9.103.3.2 isPluginV2()
virtual bool nvcaffeparser1::IPluginFactoryV2::isPluginV2 (
char const ∗ layerName ) [pure virtual], [noexcept]
A user implemented function that determines if a layer configuration is provided by an IPluginV2.
Parameters
layerName
Name of the layer which the user wishes to validate.
The documentation for this class was generated from the following file:
512
Class Documentation
• NvCaffeParser.h
9.104
nvinfer1::IPluginRegistry Class Reference
Single registration point for all plugins in an application. It is used to find plugin implementations during engine
deserialization. Internally, the plugin registry is considered to be a singleton so all plugins in an application are part of
the same global registry. Note that the plugin registry is only supported for plugins of type IPluginV2 and should also
have a corresponding IPluginCreator implementation.
#include <NvInferRuntimeCommon.h>
Public Types
• using PluginLibraryHandle = void ∗
Pointer for plugin library handle.
Public Member Functions
• virtual bool registerCreator (IPluginCreator &creator, AsciiChar const ∗const pluginNamespace) noexcept=0
Register a plugin creator. Returns false if one with same type is already registered.
• virtual IPluginCreator ∗const ∗ getPluginCreatorList (int32 t ∗const numCreators) const noexcept=0
Return all the registered plugin creators and the number of registered plugin creators. Returns nullptr if none found.
• virtual IPluginCreator ∗ getPluginCreator (AsciiChar const ∗const pluginName, AsciiChar const ∗const plugin←↩
Version, AsciiChar const ∗const pluginNamespace=””) noexcept=0
Return plugin creator based on plugin name, version, and namespace associated with plugin during network creation.
• virtual void setErrorRecorder (IErrorRecorder ∗const recorder) noexcept=0
Set the ErrorRecorder for this interface.
• virtual IErrorRecorder ∗ getErrorRecorder () const noexcept=0
Get the ErrorRecorder assigned to this interface.
• virtual bool deregisterCreator (IPluginCreator const &creator) noexcept=0
Deregister a previously registered plugin creator.
• virtual bool isParentSearchEnabled () const =0
Return whether the parent registry will be searched if a plugin is not found in this registry default: true.
• virtual void setParentSearchEnabled (bool const enabled)=0
Set whether the parent registry will be searched if a plugin is not found in this registry.
• virtual PluginLibraryHandle loadLibrary (AsciiChar const ∗pluginPath) noexcept=0
Load and register a shared library of plugins.
• virtual void deregisterLibrary (PluginLibraryHandle handle) noexcept=0
Deregister plugins associated with a library. Any resources acquired when the library was loaded will be released.
Protected Member Functions
• virtual ∼IPluginRegistry () noexcept=default
9.104 nvinfer1::IPluginRegistry Class Reference
513
9.104.1
Detailed Description
Single registration point for all plugins in an application. It is used to find plugin implementations during engine
deserialization. Internally, the plugin registry is considered to be a singleton so all plugins in an application are part of
the same global registry. Note that the plugin registry is only supported for plugins of type IPluginV2 and should also
have a corresponding IPluginCreator implementation.
See also
IPluginV2 and IPluginCreator
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
In the automotive safety context, be sure to call IPluginRegistry::setErrorRecorder() to register an error
recorder with the registry before using other methods in the registry.
9.104.2
Member Typedef Documentation
9.104.2.1 PluginLibraryHandle
using nvinfer1::IPluginRegistry::PluginLibraryHandle = void∗
Pointer for plugin library handle.
9.104.3
Constructor & Destructor Documentation
9.104.3.1 ∼IPluginRegistry()
virtual nvinfer1::IPluginRegistry::∼IPluginRegistry ( ) [protected], [virtual], [default], [noexcept]
9.104.4
Member Function Documentation
514
Class Documentation
9.104.4.1 deregisterCreator()
virtual bool nvinfer1::IPluginRegistry::deregisterCreator (
IPluginCreator const & creator ) [pure virtual], [noexcept]
Deregister a previously registered plugin creator.
Since there may be a desire to limit the number of plugins, this function provides a mechanism for removing plugin
creators registered in TensorRT. The plugin creator that is specified by creator is removed from TensorRT and no
longer tracked.
Returns
True if the plugin creator was deregistered, false if it was not found in the registry or otherwise could not be
deregistered.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.104.4.2 deregisterLibrary()
virtual void nvinfer1::IPluginRegistry::deregisterLibrary (
PluginLibraryHandle handle ) [pure virtual], [noexcept]
Deregister plugins associated with a library. Any resources acquired when the library was loaded will be released.
Parameters
handle
the plugin library handle to deregister.
9.104.4.3 getErrorRecorder()
virtual IErrorRecorder ∗ nvinfer1::IPluginRegistry::getErrorRecorder ( ) const [pure virtual],
[noexcept]
Get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A default error recorder does not exist, so a nullptr will
be returned if setErrorRecorder has not been called, or an ErrorRecorder has not been inherited.
9.104 nvinfer1::IPluginRegistry Class Reference
515
Returns
A pointer to the IErrorRecorder object that has been registered.
See also
setErrorRecorder()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.104.4.4 getPluginCreator()
virtual IPluginCreator ∗ nvinfer1::IPluginRegistry::getPluginCreator (
AsciiChar const ∗const pluginName,
AsciiChar const ∗const pluginVersion,
AsciiChar const ∗const pluginNamespace = "" ) [pure virtual], [noexcept]
Return plugin creator based on plugin name, version, and namespace associated with plugin during network creation.
Warning
The strings pluginName, pluginVersion, and pluginNamespace must be 1024 bytes or less including the
NULL terminator and must be NULL terminated.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.104.4.5 getPluginCreatorList()
virtual IPluginCreator ∗const ∗ nvinfer1::IPluginRegistry::getPluginCreatorList (
int32 t ∗const numCreators ) const [pure virtual], [noexcept]
Return all the registered plugin creators and the number of registered plugin creators. Returns nullptr if none found.
Usage considerations
• Allowed context for the API call
- Thread-safe: No
516
Class Documentation
9.104.4.6 isParentSearchEnabled()
virtual bool nvinfer1::IPluginRegistry::isParentSearchEnabled ( ) const [pure virtual]
Return whether the parent registry will be searched if a plugin is not found in this registry default: true.
Returns
bool variable indicating whether parent search is enabled.
See also
setParentSearchEnabled
9.104.4.7 loadLibrary()
virtual PluginLibraryHandle nvinfer1::IPluginRegistry::loadLibrary (
AsciiChar const ∗ pluginPath ) [pure virtual], [noexcept]
Load and register a shared library of plugins.
Parameters
pluginPath
the plugin library path.
Returns
The loaded plugin library handle. The call will fail and return nullptr if any of the plugins are already registered.
9.104.4.8 registerCreator()
virtual bool nvinfer1::IPluginRegistry::registerCreator (
IPluginCreator & creator,
AsciiChar const ∗const pluginNamespace ) [pure virtual], [noexcept]
Register a plugin creator. Returns false if one with same type is already registered.
Warning
The string pluginNamespace must be 1024 bytes or less including the NULL terminator and must be NULL
terminated.
9.104 nvinfer1::IPluginRegistry Class Reference
517
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes; calls to this method will be synchronized by a mutex.
9.104.4.9 setErrorRecorder()
virtual void nvinfer1::IPluginRegistry::setErrorRecorder (
IErrorRecorder ∗const recorder ) [pure virtual], [noexcept]
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function
will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder
with the interface, resulting in a call to decRefCount if a recorder has been registered.
Parameters
recorder
The error recorder to register with this interface.
See also
getErrorRecorder()
Usage considerations
• Allowed context for the API call
- Thread-safe: No
9.104.4.10 setParentSearchEnabled()
virtual void nvinfer1::IPluginRegistry::setParentSearchEnabled (
bool const enabled ) [pure virtual]
Set whether the parent registry will be searched if a plugin is not found in this registry.
518
Class Documentation
Parameters
enabled
The bool variable indicating whether parent search is enabled.
See also
isParentSearchEnabled
The documentation for this class was generated from the following file:
• NvInferRuntimeCommon.h
9.105
nvinfer1::safe::IPluginRegistry Class Reference
Single registration point for all plugins in an application. It is used to find plugin implementations during engine
deserialization. Internally, the plugin registry is considered to be a singleton so all plugins in an application are part of
the same global registry. Note that the plugin registry is only supported for plugins of type IPluginV2 and should also
have a corresponding IPluginCreator implementation.
#include <NvInferSafeRuntime.h>
Public Member Functions
• virtual bool registerCreator (IPluginCreator &creator, AsciiChar const ∗const pluginNamespace) noexcept=0
Register a plugin creator. Returns false if one with same type is already registered.
• virtual IPluginCreator ∗const ∗ getPluginCreatorList (int32 t ∗const numCreators) const noexcept=0
Return all the registered plugin creators and the number of registered plugin creators. Returns nullptr if none found.
• virtual IPluginCreator ∗ getPluginCreator (AsciiChar const ∗const pluginName, AsciiChar const ∗const plugin←↩
Version, AsciiChar const ∗const pluginNamespace=””) noexcept=0
Return plugin creator based on plugin name, version, and namespace associated with plugin during network creation.
• virtual void setErrorRecorder (IErrorRecorder ∗const recorder) noexcept=0
Set the ErrorRecorder for this interface.
• virtual IErrorRecorder ∗ getErrorRecorder () const noexcept=0
Get the ErrorRecorder assigned to this interface.
• virtual bool deregisterCreator (IPluginCreator const &creator) noexcept=0
Deregister a previously registered plugin creator.
Protected Member Functions
• virtual ∼IPluginRegistry () noexcept=default
9.105 nvinfer1::safe::IPluginRegistry Class Reference
519
9.105.1
Detailed Description
Single registration point for all plugins in an application. It is used to find plugin implementations during engine
deserialization. Internally, the plugin registry is considered to be a singleton so all plugins in an application are part of
the same global registry. Note that the plugin registry is only supported for plugins of type IPluginV2 and should also
have a corresponding IPluginCreator implementation.
See also
IPluginV2 and IPluginCreator
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
IPluginRegistry::setErrorRecorder() must be called to register an error recorder with the registry before
using other methods in the registry.
9.105.2
Constructor & Destructor Documentation
9.105.2.1 ∼IPluginRegistry()
virtual nvinfer1::safe::IPluginRegistry::∼IPluginRegistry ( ) [protected], [virtual], [default],
[noexcept]
9.105.3
Member Function Documentation
9.105.3.1 deregisterCreator()
virtual bool nvinfer1::safe::IPluginRegistry::deregisterCreator (
IPluginCreator const & creator ) [pure virtual], [noexcept]
Deregister a previously registered plugin creator.
Since there may be a desire to limit the number of plugins, this function provides a mechanism for removing plugin
creators registered in TensorRT. The plugin creator that is specified by creator is removed from TensorRT and no
longer tracked.
Returns
True if the plugin creator was deregistered, false if it was not found in the registry or otherwise could not be
deregistered.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
520
Class Documentation
9.105.3.2 getErrorRecorder()
virtual IErrorRecorder ∗ nvinfer1::safe::IPluginRegistry::getErrorRecorder ( ) const [pure virtual],
[noexcept]
Get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A default error recorder does not exist, so a nullptr will
be returned if setErrorRecorder has not been called, or an ErrorRecorder has not been inherited.
Returns
A pointer to the IErrorRecorder object that has been registered.
See also
setErrorRecorder()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.105.3.3 getPluginCreator()
virtual IPluginCreator ∗ nvinfer1::safe::IPluginRegistry::getPluginCreator (
AsciiChar const ∗const pluginName,
AsciiChar const ∗const pluginVersion,
AsciiChar const ∗const pluginNamespace = "" ) [pure virtual], [noexcept]
Return plugin creator based on plugin name, version, and namespace associated with plugin during network creation.
Warning
The strings pluginName, pluginVersion, and pluginNamespace must be 1024 bytes or less including the
NULL terminator and must be NULL terminated.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.105 nvinfer1::safe::IPluginRegistry Class Reference
521
9.105.3.4 getPluginCreatorList()
virtual IPluginCreator ∗const ∗ nvinfer1::safe::IPluginRegistry::getPluginCreatorList (
int32 t ∗const numCreators ) const [pure virtual], [noexcept]
Return all the registered plugin creators and the number of registered plugin creators. Returns nullptr if none found.
Usage considerations
• Allowed context for the API call
- Thread-safe: No
9.105.3.5 registerCreator()
virtual bool nvinfer1::safe::IPluginRegistry::registerCreator (
IPluginCreator & creator,
AsciiChar const ∗const pluginNamespace ) [pure virtual], [noexcept]
Register a plugin creator. Returns false if one with same type is already registered.
Warning
The string pluginNamespace must be 1024 bytes or less including the NULL terminator and must be NULL
terminated.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes; calls to this method will be synchronized by a mutex.
9.105.3.6 setErrorRecorder()
virtual void nvinfer1::safe::IPluginRegistry::setErrorRecorder (
IErrorRecorder ∗const recorder ) [pure virtual], [noexcept]
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function
will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder
with the interface, resulting in a call to decRefCount if a recorder has been registered.
522
Class Documentation
Parameters
recorder
The error recorder to register with this interface.
See also
getErrorRecorder()
Usage considerations
• Allowed context for the API call
- Thread-safe: No
The documentation for this class was generated from the following file:
• NvInferSafeRuntime.h
9.106
nvinfer1::IPluginV2 Class Reference
Plugin class for user-implemented layers.
#include <NvInferRuntimePlugin.h>
Inheritance diagram for nvinfer1::IPluginV2:
nvinfer1::IPluginV2
nvinfer1::IPluginV2Ext
nvinfer1::IPluginV2DynamicExt
nvinfer1::IPluginV2IOExt
Public Member Functions
• virtual int32 t getTensorRTVersion () const noexcept
Return the API version with which this plugin was built.
• virtual AsciiChar const ∗ getPluginType () const noexcept=0
Return the plugin type. Should match the plugin name returned by the corresponding plugin creator.
• virtual AsciiChar const ∗ getPluginVersion () const noexcept=0
Return the plugin version. Should match the plugin version returned by the corresponding plugin creator.
• virtual int32 t getNbOutputs () const noexcept=0
Get the number of outputs from the layer.
9.106 nvinfer1::IPluginV2 Class Reference
523
•
virtual Dims getOutputDimensions (int32 t index, Dims const ∗inputs, int32 t nbInputDims) noexcept=0
Get the dimension of an output tensor.
•
virtual bool supportsFormat (DataType type, PluginFormat format) const noexcept=0
Check format support.
•
virtual void configureWithFormat (Dims const ∗inputDims, int32 t nbInputs, Dims const ∗outputDims, int32 t
nbOutputs, DataType type, PluginFormat format, int32 t maxBatchSize) noexcept=0
Configure the layer.
•
virtual int32 t initialize () noexcept=0
Initialize the layer for execution. This is called when the engine is created.
•
virtual void terminate () noexcept=0
Release resources acquired during plugin layer initialization. This is called when the engine is destroyed.
•
virtual size t getWorkspaceSize (int32 t maxBatchSize) const noexcept=0
Find the workspace size required by the layer.
•
virtual int32 t enqueue (int32 t batchSize, void const ∗const ∗inputs, void ∗const ∗outputs, void ∗workspace,
cudaStream t stream) noexcept=0
Execute the layer.
•
virtual size t getSerializationSize () const noexcept=0
Find the size of the serialization buffer required.
•
virtual void serialize (void ∗buffer) const noexcept=0
Serialize the layer.
•
virtual void destroy () noexcept=0
Destroy the plugin object. This will be called when the network, builder or engine is destroyed.
•
virtual IPluginV2 ∗ clone () const noexcept=0
Clone the plugin object. This copies over internal plugin parameters and returns a new plugin object with these param-
eters.
•
virtual void setPluginNamespace (AsciiChar const ∗pluginNamespace) noexcept=0
Set the namespace that this plugin object belongs to. Ideally, all plugin objects from the same plugin library should have
the same namespace.
•
virtual AsciiChar const ∗ getPluginNamespace () const noexcept=0
Return the namespace of the plugin object.
9.106.1
Detailed Description
Plugin class for user-implemented layers.
Plugins are a mechanism for applications to implement custom layers. When combined with IPluginCreator it provides
a mechanism to register plugins and look up the Plugin Registry during de-serialization.
See also
IPluginCreator
IPluginRegistry
Deprecated Deprecated in TensorRT 8.5. Implement IPluginV2DynamicExt or IPluginV2IOExt depending on your
requirement.
524
Class Documentation
9.106.2
Member Function Documentation
9.106.2.1 clone()
virtual IPluginV2 ∗ nvinfer1::IPluginV2::clone ( ) const [pure virtual], [noexcept]
Clone the plugin object. This copies over internal plugin parameters and returns a new plugin object with these
parameters.
The TensorRT runtime calls clone() to clone the plugin when an execution context is created for an engine, after the
engine has been created. The runtime does not call initialize() on the cloned plugin, so the cloned plugin should be
created in an initialized state.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin or when creating multiple execution con-
texts.
Implemented in nvinfer1::IPluginV2DynamicExt, and nvinfer1::IPluginV2Ext.
9.106.2.2 configureWithFormat()
virtual void nvinfer1::IPluginV2::configureWithFormat (
Dims const ∗ inputDims,
int32 t nbInputs,
Dims const ∗ outputDims,
int32 t nbOutputs,
DataType type,
PluginFormat format,
int32 t maxBatchSize ) [pure virtual], [noexcept]
Configure the layer.
This function is called by the builder prior to initialize(). It provides an opportunity for the layer to make algorithm
choices on the basis of its weights, dimensions, and maximum batch size.
Parameters
inputDims
The input tensor dimensions.
nbInputs
The number of inputs.
outputDims
The output tensor dimensions.
nbOutputs
The number of outputs.
type
The data type selected for the engine.
format
The format selected for the engine.
maxBatchSize
The maximum batch size.
9.106 nvinfer1::IPluginV2 Class Reference
525
The dimensions passed here do not include the outermost batch size (i.e. for 2-D image networks, they will be 3-
dimensional CHW dimensions).
Warning
for the format field, the values PluginFormat::kCHW4, PluginFormat::kCHW16, and PluginFormat::k←↩
CHW32 will not be passed in, this is to keep backward compatibility with TensorRT 5.x series. Use Plugin←↩
V2IOExt or PluginV2DynamicExt for other PluginFormats.
DataType:kBOOL and DataType::kUINT8 are not supported.
See also
clone()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin. However, TensorRT will not call this
method from two threads simultaneously on a given clone of a plugin.
Implemented in nvinfer1::IPluginV2Ext.
9.106.2.3 destroy()
virtual void nvinfer1::IPluginV2::destroy ( ) [pure virtual], [noexcept]
Destroy the plugin object. This will be called when the network, builder or engine is destroyed.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.106.2.4 enqueue()
virtual int32 t nvinfer1::IPluginV2::enqueue (
int32 t batchSize,
void const ∗const ∗ inputs,
void ∗const ∗ outputs,
void ∗ workspace,
cudaStream t stream ) [pure virtual], [noexcept]
Execute the layer.
526
Class Documentation
Parameters
batchSize
The number of inputs in the batch.
inputs
The memory for the input tensors.
outputs
The memory for the output tensors.
workspace
Workspace for execution.
stream
The stream in which to execute the kernels.
Returns
0 for success, else non-zero (which will cause engine termination).
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
multiple execution contexts are used during runtime.
9.106.2.5 getNbOutputs()
virtual int32 t nvinfer1::IPluginV2::getNbOutputs ( ) const [pure virtual], [noexcept]
Get the number of outputs from the layer.
Returns
The number of outputs.
This function is called by the implementations of INetworkDefinition and IBuilder. In particular, it is called prior to
any call to initialize().
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.106.2.6 getOutputDimensions()
virtual Dims nvinfer1::IPluginV2::getOutputDimensions (
int32 t index,
Dims const ∗ inputs,
int32 t nbInputDims ) [pure virtual], [noexcept]
Get the dimension of an output tensor.
9.106 nvinfer1::IPluginV2 Class Reference
527
Parameters
index
The index of the output tensor.
inputs
The input tensors.
nbInputDims
The number of input tensors.
This function is called by the implementations of INetworkDefinition and IBuilder. In particular, it is called prior to
any call to initialize().
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
Note
In any non-IPluginV2DynamicExt plugin, batch size should not be included in the returned dimensions, even
if the plugin is expected to be run in a network with explicit batch mode enabled. Please see the TensorRT
Developer Guide for more details on how plugin inputs and outputs behave.
9.106.2.7 getPluginNamespace()
virtual AsciiChar const ∗ nvinfer1::IPluginV2::getPluginNamespace ( ) const [pure virtual], [noexcept]
Return the namespace of the plugin object.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.106.2.8 getPluginType()
virtual AsciiChar const ∗ nvinfer1::IPluginV2::getPluginType ( ) const [pure virtual], [noexcept]
Return the plugin type. Should match the plugin name returned by the corresponding plugin creator.
See also
IPluginCreator::getPluginName()
528
Class Documentation
Warning
The string returned must be 1024 bytes or less including the NULL terminator and must be NULL termi-
nated.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.106.2.9 getPluginVersion()
virtual AsciiChar const ∗ nvinfer1::IPluginV2::getPluginVersion ( ) const [pure virtual], [noexcept]
Return the plugin version. Should match the plugin version returned by the corresponding plugin creator.
See also
IPluginCreator::getPluginVersion()
Warning
The string returned must be 1024 bytes or less including the NULL terminator and must be NULL termi-
nated.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.106.2.10 getSerializationSize()
virtual size t nvinfer1::IPluginV2::getSerializationSize ( ) const [pure virtual], [noexcept]
Find the size of the serialization buffer required.
Returns
The size of the serialization buffer.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.106 nvinfer1::IPluginV2 Class Reference
529
9.106.2.11 getTensorRTVersion()
virtual int32 t nvinfer1::IPluginV2::getTensorRTVersion ( ) const [inline], [virtual], [noexcept]
Return the API version with which this plugin was built.
Do not override this method as it is used by the TensorRT library to maintain backwards-compatibility with plugins.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, the implementation provided here is safe to call from any thread.
Reimplemented in nvinfer1::IPluginV2DynamicExt, nvinfer1::IPluginV2Ext, and nvinfer1::IPluginV2IOExt.
9.106.2.12 getWorkspaceSize()
virtual size t nvinfer1::IPluginV2::getWorkspaceSize (
int32 t maxBatchSize ) const [pure virtual], [noexcept]
Find the workspace size required by the layer.
This function is called during engine startup, after initialize(). The workspace size returned should be sufficient for any
batch size up to the maximum.
Returns
The workspace size.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin. However, TensorRT will not call this
method from two threads simultaneously on a given clone of a plugin.
530
Class Documentation
9.106.2.13 initialize()
virtual int32 t nvinfer1::IPluginV2::initialize ( ) [pure virtual], [noexcept]
Initialize the layer for execution. This is called when the engine is created.
Returns
0 for success, else non-zero (which will cause engine termination).
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin or when using multiple execution contexts
using this plugin.
9.106.2.14 serialize()
virtual void nvinfer1::IPluginV2::serialize (
void ∗ buffer ) const [pure virtual], [noexcept]
Serialize the layer.
Parameters
buffer
A pointer to a buffer to serialize data. Size of buffer must be equal to value returned by getSerializationSize.
See also
getSerializationSize()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.106 nvinfer1::IPluginV2 Class Reference
531
9.106.2.15 setPluginNamespace()
virtual void nvinfer1::IPluginV2::setPluginNamespace (
AsciiChar const ∗ pluginNamespace ) [pure virtual], [noexcept]
Set the namespace that this plugin object belongs to. Ideally, all plugin objects from the same plugin library should
have the same namespace.
Parameters
pluginNamespace
The namespace for the plugin object.
Warning
The string pluginNamespace must be 1024 bytes or less including the NULL terminator and must be NULL
terminated.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.106.2.16 supportsFormat()
virtual bool nvinfer1::IPluginV2::supportsFormat (
DataType type,
PluginFormat format ) const [pure virtual], [noexcept]
Check format support.
Parameters
type
DataType requested.
format
PluginFormat requested.
Returns
true if the plugin supports the type-format combination.
This function is called by the implementations of INetworkDefinition, IBuilder, and safe::ICudaEngine/ICudaEngine.
In particular, it is called when creating an engine and when deserializing an engine.
532
Class Documentation
Warning
for the format field, the values PluginFormat::kCHW4, PluginFormat::kCHW16, and PluginFormat::k←↩
CHW32 will not be passed in, this is to keep backward compatibility with TensorRT 5.x series. Use Plugin←↩
V2IOExt or PluginV2DynamicExt for other PluginFormats.
DataType:kBOOL and DataType::kUINT8 are not supported.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.106.2.17 terminate()
virtual void nvinfer1::IPluginV2::terminate ( ) [pure virtual], [noexcept]
Release resources acquired during plugin layer initialization. This is called when the engine is destroyed.
See also
initialize()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin or when using multiple execution contexts
using this plugin. However, TensorRT will not call this method from two threads simultaneously on a given
clone of a plugin.
The documentation for this class was generated from the following file:
• NvInferRuntimePlugin.h
9.107 nvinfer1::IPluginV2DynamicExt Class Reference
533
9.107
nvinfer1::IPluginV2DynamicExt Class Reference
#include <NvInferRuntime.h>
Inheritance diagram for nvinfer1::IPluginV2DynamicExt:
nvinfer1::IPluginV2
nvinfer1::IPluginV2Ext
nvinfer1::IPluginV2DynamicExt
Public Member Functions
• IPluginV2DynamicExt ∗ clone () const noexcept override=0
Clone the plugin object. This copies over internal plugin parameters as well and returns a new plugin object with
these parameters. If the source plugin is pre-configured with configurePlugin(), the returned object should also be pre-
configured. The returned object should allow attachToContext() with a new execution context. Cloned plugin objects
can share the same per-engine immutable resource (e.g. weights) with the source object (e.g. via ref-counting) to avoid
duplication.
• virtual DimsExprs getOutputDimensions (int32 t outputIndex, DimsExprs const ∗inputs, int32 t nbInputs,
IExprBuilder &exprBuilder) noexcept=0
Get expressions for computing dimensions of an output tensor from dimensions of the input tensors.
• virtual bool supportsFormatCombination (int32 t pos, PluginTensorDesc const ∗inOut, int32 t nbInputs, int32 t
nbOutputs) noexcept=0
Return true if plugin supports the format and datatype for the input/output indexed by pos.
• virtual void configurePlugin (DynamicPluginTensorDesc const ∗in, int32 t nbInputs, DynamicPluginTensorDesc
const ∗out, int32 t nbOutputs) noexcept=0
Configure the plugin.
• virtual size t getWorkspaceSize (PluginTensorDesc const ∗inputs, int32 t nbInputs, PluginTensorDesc const
∗outputs, int32 t nbOutputs) const noexcept=0
Find the workspace size required by the layer.
• virtual int32 t enqueue (PluginTensorDesc const ∗inputDesc, PluginTensorDesc const ∗outputDesc, void const
∗const ∗inputs, void ∗const ∗outputs, void ∗workspace, cudaStream t stream) noexcept=0
Execute the layer.
Static Public Attributes
• static constexpr int32 t kFORMAT COMBINATION LIMIT = 100
Protected Member Functions
• int32 t getTensorRTVersion () const noexcept override
Return the API version with which this plugin was built. The upper byte reserved by TensorRT and is used to differentiate
this from IPluginV2.
• virtual ∼IPluginV2DynamicExt () noexcept
534
Class Documentation
9.107.1
Detailed Description
Similar to IPluginV2Ext, but with support for dynamic shapes.
Clients should override the public methods, including the following inherited methods:
virtual int32_t getNbOutputs() const noexcept = 0;
virtual nvinfer1::DataType getOutputDataType(int32_t index, nvinfer1::DataType const* inputTypes, int32_t
nbInputs) const noexcept = 0; virtual size_t getSerializationSize() const noexcept = 0; virtual void
serialize(void* buffer) const noexcept = 0; virtual void destroy() noexcept = 0; virtual void
setPluginNamespace(char const* pluginNamespace) noexcept = 0; virtual char const* getPluginNamespace() const
noexcept = 0;
For getOutputDataType, the inputTypes will always be DataType::kFLOAT or DataType::kINT32, and the returned
type is canonicalized to DataType::kFLOAT if it is DataType::kHALF or DataType:kINT8. Details about the floating-
point precision are elicited later by method supportsFormatCombination.
9.107.2
Constructor & Destructor Documentation
9.107.2.1 ∼IPluginV2DynamicExt()
virtual nvinfer1::IPluginV2DynamicExt::∼IPluginV2DynamicExt ( ) [inline], [protected], [virtual],
[noexcept]
9.107.3
Member Function Documentation
9.107.3.1 clone()
IPluginV2DynamicExt ∗ nvinfer1::IPluginV2DynamicExt::clone ( ) const [override], [pure virtual],
[noexcept]
Clone the plugin object. This copies over internal plugin parameters as well and returns a new plugin object with
these parameters. If the source plugin is pre-configured with configurePlugin(), the returned object should also be pre-
configured. The returned object should allow attachToContext() with a new execution context. Cloned plugin objects
can share the same per-engine immutable resource (e.g. weights) with the source object (e.g. via ref-counting) to avoid
duplication.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
Implements nvinfer1::IPluginV2Ext.
9.107 nvinfer1::IPluginV2DynamicExt Class Reference
535
9.107.3.2 configurePlugin()
virtual void nvinfer1::IPluginV2DynamicExt::configurePlugin (
DynamicPluginTensorDesc const ∗ in,
int32 t nbInputs,
DynamicPluginTensorDesc const ∗ out,
int32 t nbOutputs ) [pure virtual], [noexcept]
Configure the plugin.
configurePlugin() can be called multiple times in both the build and execution phases. The build phase happens
before initialize() is called and only occurs during creation of an engine by IBuilder. The execution phase happens
after initialize() is called and occurs during both creation of an engine by IBuilder and execution of an engine by
IExecutionContext.
Build phase: IPluginV2DynamicExt->configurePlugin is called when a plugin is being prepared for profiling but
not for any specific input size. This provides an opportunity for the plugin to make algorithmic choices on the
basis of input and output formats, along with the bound of possible dimensions. The min and max value of the
DynamicPluginTensorDesc correspond to the kMIN and kMAX value of the current profile that the plugin is being
profiled for, with the desc.dims field corresponding to the dimensions of plugin specified at network creation. Wildcard
dimensions will exist during this phase in the desc.dims field.
Execution phase: IPluginV2DynamicExt->configurePlugin is called when a plugin is being prepared for executing the
plugin for a specific dimensions. This provides an opportunity for the plugin to change algorithmic choices based on
the explicit input dimensions stored in desc.dims field.
• IBuilder will call this function once per profile, with desc.dims resolved to the values specified by the kOPT field
of the current profile. Wildcard dimensions will not exist during this phase.
• IExecutionContext will call this during the next subsequent instance enqueue[V2]() or execute[V2]() if:
- The batch size is changed from previous call of execute()/enqueue() if hasImplicitBatchDimension() returns
true.
- The optimization profile is changed via setOptimizationProfile() or setOptimizationProfileAsync().
- An input shape binding is changed via setInputShapeBinding().
- An input execution binding is changed via setBindingDimensions().
Warning
The execution phase is timing critical during IExecutionContext but is not part of the timing
loop when called from IBuilder. Performance bottlenecks of configurePlugin won't show up
during engine building but will be visible during execution after calling functions that trigger
layer resource updates.
Parameters
in
The input tensors attributes that are used for configuration.
nbInputs
Number of input tensors.
out
The output tensors attributes that are used for configuration.
nbOutputs
Number of output tensors.
536
Class Documentation
9.107.3.3 enqueue()
virtual int32 t nvinfer1::IPluginV2DynamicExt::enqueue (
PluginTensorDesc const ∗ inputDesc,
PluginTensorDesc const ∗ outputDesc,
void const ∗const ∗ inputs,
void ∗const ∗ outputs,
void ∗ workspace,
cudaStream t stream ) [pure virtual], [noexcept]
Execute the layer.
Parameters
inputDesc
how to interpret the memory for the input tensors.
outputDesc
how to interpret the memory for the output tensors.
inputs
The memory for the input tensors.
outputs
The memory for the output tensors.
workspace
Workspace for execution.
stream
The stream in which to execute the kernels.
Returns
0 for success, else non-zero (which will cause engine termination).
9.107.3.4 getOutputDimensions()
virtual DimsExprs nvinfer1::IPluginV2DynamicExt::getOutputDimensions (
int32 t outputIndex,
DimsExprs const ∗ inputs,
int32 t nbInputs,
IExprBuilder & exprBuilder ) [pure virtual], [noexcept]
Get expressions for computing dimensions of an output tensor from dimensions of the input tensors.
Parameters
outputIndex
The index of the output tensor
inputs
Expressions for dimensions of the input tensors
nbInputs
The number of input tensors
exprBuilder
Object for generating new expressions
9.107 nvinfer1::IPluginV2DynamicExt Class Reference
537
This function is called by the implementations of IBuilder during analysis of the network.
Example #1: A plugin has a single output that transposes the last two dimensions of the plugin's single input. The body
of the override of getOutputDimensions can be:
DimsExprs output(inputs[0]);
std::swap(output.d[output.nbDims-1], output.d[output.nbDims-2]);
return output;
Example #2: A plugin concatenates its two inputs along the first dimension. The body of the override of getOutput←↩
Dimensions can be:
DimsExprs output(inputs[0]);
output.d[0] = exprBuilder.operation(DimensionOperation::kSUM, *inputs[0].d[0], *inputs[1].d[0]);
return output;
9.107.3.5 getTensorRTVersion()
int32 t nvinfer1::IPluginV2DynamicExt::getTensorRTVersion ( ) const [inline], [override], [protected],
[virtual], [noexcept]
Return the API version with which this plugin was built. The upper byte reserved by TensorRT and is used to differen-
tiate this from IPluginV2.
Do not override this method as it is used by the TensorRT library to maintain backwards-compatibility with plugins.
Reimplemented from nvinfer1::IPluginV2.
9.107.3.6 getWorkspaceSize()
virtual size t nvinfer1::IPluginV2DynamicExt::getWorkspaceSize (
PluginTensorDesc const ∗ inputs,
int32 t nbInputs,
PluginTensorDesc const ∗ outputs,
int32 t nbOutputs ) const [pure virtual], [noexcept]
Find the workspace size required by the layer.
This function is called after the plugin is configured, and possibly during execution. The result should be a sufficient
workspace size to deal with inputs and outputs of the given size or any smaller problem.
Returns
The workspace size.
538
Class Documentation
9.107.3.7 supportsFormatCombination()
virtual bool nvinfer1::IPluginV2DynamicExt::supportsFormatCombination (
int32 t pos,
PluginTensorDesc const ∗ inOut,
int32 t nbInputs,
int32 t nbOutputs ) [pure virtual], [noexcept]
Return true if plugin supports the format and datatype for the input/output indexed by pos.
For this method inputs are numbered 0..(nbInputs-1) and outputs are numbered nbInputs..(nbInputs+nbOutputs-1).
Using this numbering, pos is an index into InOut, where 0 <= pos < nbInputs+nbOutputs.
TensorRT invokes this method to ask if the input/output indexed by pos supports the format/datatype specified by
inOut[pos].format and inOut[pos].type. The override should return true if that format/datatype at inOut[pos] are sup-
ported by the plugin. If support is conditional on other input/output formats/datatypes, the plugin can make its result
conditional on the formats/datatypes in inOut[0..pos-1], which will be set to values that the plugin supports. The
override should not inspect inOut[pos+1..nbInputs+nbOutputs-1], which will have invalid values. In other words, the
decision for pos must be based on inOut[0..pos] only.
Some examples:
• A definition for a plugin that supports only FP16 NCHW:
return inOut.format[pos] == TensorFormat::kLINEAR && inOut.type[pos] == DataType::kHALF;
• A definition for a plugin that supports only FP16 NCHW for its two inputs, and FP32 NCHW for its single
output:
return inOut.format[pos] == TensorFormat::kLINEAR && (inOut.type[pos] == (pos < 2 ? DataType::kHALF :
DataType::kFLOAT));
• A definition for a ”polymorphic” plugin with two inputs and one output that supports any format or type, but the
inputs and output must have the same format and type:
return pos == 0 || (inOut.format[pos] == inOut.format[0] && inOut.type[pos] == inOut.type[0]);
Warning: TensorRT will stop asking for formats once it finds kFORMAT COMBINATION LIMIT on combina-
tions.
9.107.4
Member Data Documentation
9.107.4.1 kFORMAT COMBINATION LIMIT
constexpr int32 t nvinfer1::IPluginV2DynamicExt::kFORMAT COMBINATION LIMIT = 100 [static], [constexpr]
Limit on number of format combinations accepted.
The documentation for this class was generated from the following file:
• NvInferRuntime.h
9.108 nvinfer1::IPluginV2Ext Class Reference
539
9.108
nvinfer1::IPluginV2Ext Class Reference
Plugin class for user-implemented layers.
#include <NvInferRuntimePlugin.h>
Inheritance diagram for nvinfer1::IPluginV2Ext:
nvinfer1::IPluginV2
nvinfer1::IPluginV2Ext
nvinfer1::IPluginV2DynamicExt
nvinfer1::IPluginV2IOExt
Public Member Functions
• virtual nvinfer1::DataType getOutputDataType (int32 t index, nvinfer1::DataType const ∗inputTypes, int32 ←↩
t nbInputs) const noexcept=0
Return the DataType of the plugin output at the requested index.
• virtual bool isOutputBroadcastAcrossBatch (int32 t outputIndex, bool const ∗inputIsBroadcasted, int32 t nb←↩
Inputs) const noexcept=0
Return true if output tensor is broadcast across a batch.
• virtual bool canBroadcastInputAcrossBatch (int32 t inputIndex) const noexcept=0
Return true if plugin can use input that is broadcast across batch without replication.
• virtual void configurePlugin (Dims const ∗inputDims, int32 t nbInputs, Dims const ∗outputDims, int32 t nb←↩
Outputs, DataType const ∗inputTypes, DataType const ∗outputTypes, bool const ∗inputIsBroadcast, bool const
∗outputIsBroadcast, PluginFormat floatFormat, int32 t maxBatchSize) noexcept=0
Configure the layer with input and output data types.
• IPluginV2Ext ()=default
• ∼IPluginV2Ext () override=default
• virtual void attachToContext (cudnnContext ∗, cublasContext ∗, IGpuAllocator ∗) noexcept
Attach the plugin object to an execution context and grant the plugin the access to some context resource.
• virtual void detachFromContext () noexcept
Detach the plugin object from its execution context.
• IPluginV2Ext ∗ clone () const noexcept override=0
Clone the plugin object. This copies over internal plugin parameters as well and returns a new plugin object with
these parameters. If the source plugin is pre-configured with configurePlugin(), the returned object should also be pre-
configured. The returned object should allow attachToContext() with a new execution context. Cloned plugin objects
can share the same per-engine immutable resource (e.g. weights) with the source object (e.g. via ref-counting) to avoid
duplication.
Protected Member Functions
• int32 t getTensorRTVersion () const noexcept override
Return the API version with which this plugin was built. The upper byte reserved by TensorRT and is used to differentiate
this from IPluginV2.
• void configureWithFormat (Dims const ∗, int32 t, Dims const ∗, int32 t, DataType, PluginFormat, int32 ←↩
t) noexcept override
Derived classes should not implement this. In a C++11 API it would be override final.
540
Class Documentation
9.108.1
Detailed Description
Plugin class for user-implemented layers.
Plugins are a mechanism for applications to implement custom layers. This interface provides additional capabilities
to the IPluginV2 interface by supporting different output data types and broadcast across batch.
See also
IPluginV2
Deprecated Deprecated in TensorRT 8.5. Implement IPluginV2DynamicExt or IPluginV2IOExt depending on your
requirement.
9.108.2
Constructor & Destructor Documentation
9.108.2.1 IPluginV2Ext()
nvinfer1::IPluginV2Ext::IPluginV2Ext ( ) [default]
9.108.2.2 ∼IPluginV2Ext()
nvinfer1::IPluginV2Ext::∼IPluginV2Ext ( ) [override], [default]
9.108.3
Member Function Documentation
9.108.3.1 attachToContext()
virtual void nvinfer1::IPluginV2Ext::attachToContext (
cudnnContext ∗ ,
cublasContext ∗ ,
IGpuAllocator ∗
) [inline], [virtual], [noexcept]
Attach the plugin object to an execution context and grant the plugin the access to some context resource.
9.108 nvinfer1::IPluginV2Ext Class Reference
541
Parameters
cudnn
The CUDNN context handle of the execution context
cublas
The cublas context handle of the execution context
allocator
The allocator used by the execution context
This function is called automatically for each plugin when a new execution context is created. If the context was
created without resources, this method is not called until the resources are assigned. It is also called if new resources
are assigned to the context.
If the plugin needs per-context resource, it can be allocated here. The plugin can also get context-owned CUDNN and
CUBLAS context here.
Note
In the automotive safety context, the CUDNN and CUBLAS parameters will be nullptr because CUDNN and
CUBLAS is not used by the safe runtime.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.108.3.2 canBroadcastInputAcrossBatch()
virtual bool nvinfer1::IPluginV2Ext::canBroadcastInputAcrossBatch (
int32 t inputIndex ) const [pure virtual], [noexcept]
Return true if plugin can use input that is broadcast across batch without replication.
Parameters
inputIndex
Index of input that could be broadcast.
For each input whose tensor is semantically broadcast across a batch, TensorRT calls this method before calling
configurePlugin. If canBroadcastInputAcrossBatch returns true, TensorRT will not replicate the input tensor; i.e.,
there will be a single copy that the plugin should share across the batch. If it returns false, TensorRT will replicate the
input tensor so that it appears like a non-broadcasted tensor.
This method is called only for inputs that can be broadcast.
Usage considerations
542
Class Documentation
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.108.3.3 clone()
IPluginV2Ext ∗ nvinfer1::IPluginV2Ext::clone ( ) const [override], [pure virtual], [noexcept]
Clone the plugin object. This copies over internal plugin parameters as well and returns a new plugin object with
these parameters. If the source plugin is pre-configured with configurePlugin(), the returned object should also be pre-
configured. The returned object should allow attachToContext() with a new execution context. Cloned plugin objects
can share the same per-engine immutable resource (e.g. weights) with the source object (e.g. via ref-counting) to avoid
duplication.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
Implements nvinfer1::IPluginV2.
Implemented in nvinfer1::IPluginV2DynamicExt.
9.108.3.4 configurePlugin()
virtual void nvinfer1::IPluginV2Ext::configurePlugin (
Dims const ∗ inputDims,
int32 t nbInputs,
Dims const ∗ outputDims,
int32 t nbOutputs,
DataType const ∗ inputTypes,
DataType const ∗ outputTypes,
bool const ∗ inputIsBroadcast,
bool const ∗ outputIsBroadcast,
PluginFormat floatFormat,
int32 t maxBatchSize ) [pure virtual], [noexcept]
Configure the layer with input and output data types.
This function is called by the builder prior to initialize(). It provides an opportunity for the layer to make algorithm
choices on the basis of its weights, dimensions, data types and maximum batch size.
9.108 nvinfer1::IPluginV2Ext Class Reference
543
Parameters
inputDims
The input tensor dimensions.
nbInputs
The number of inputs.
outputDims
The output tensor dimensions.
nbOutputs
The number of outputs.
inputTypes
The data types selected for the plugin inputs.
outputTypes
The data types selected for the plugin outputs.
inputIsBroadcast
True for each input that the plugin must broadcast across the batch.
outputIsBroadcast
True for each output that TensorRT will broadcast across the batch.
floatFormat
The format selected for the engine for the floating point inputs/outputs.
maxBatchSize
The maximum batch size.
The dimensions passed here do not include the outermost batch size (i.e. for 2-D image networks, they will be 3-
dimensional CHW dimensions). When inputIsBroadcast or outputIsBroadcast is true, the outermost batch size for
that input or output should be treated as if it is one. Index 'i' of inputIsBroadcast is true only if the input is semanti-
cally broadcast across the batch and calling canBroadcastInputAcrossBatch with argument 'i' returns true. Index 'i' of
outputIsBroadcast is true only if calling isOutputBroadcastAcrossBatch with argument 'i' returns true.
Warning
for the floatFormat field, the values PluginFormat::kCHW4, PluginFormat::kCHW16, and PluginFormat←↩
::kCHW32 will not be passed in, this is to keep backward compatibility with TensorRT 5.x series. Use
PluginV2IOExt or PluginV2DynamicExt for other PluginFormats.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin. However, TensorRT will not call this
method from two threads simultaneously on a given clone of a plugin.
9.108.3.5 configureWithFormat()
void nvinfer1::IPluginV2Ext::configureWithFormat (
Dims const ∗ ,
int32 t ,
Dims const ∗ ,
int32 t ,
DataType ,
PluginFormat ,
int32 t
) [inline], [override], [protected], [virtual], [noexcept]
Derived classes should not implement this. In a C++11 API it would be override final.
Implements nvinfer1::IPluginV2.
544
Class Documentation
9.108.3.6 detachFromContext()
virtual void nvinfer1::IPluginV2Ext::detachFromContext ( ) [inline], [virtual], [noexcept]
Detach the plugin object from its execution context.
This function is called automatically for each plugin when a execution context is destroyed or the context resources are
unassigned from the context.
If the plugin owns per-context resource, it can be released here.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.108.3.7 getOutputDataType()
virtual nvinfer1::DataType nvinfer1::IPluginV2Ext::getOutputDataType (
int32 t index,
nvinfer1::DataType const ∗ inputTypes,
int32 t nbInputs ) const [pure virtual], [noexcept]
Return the DataType of the plugin output at the requested index.
The default behavior should be to return the type of the first input, or DataType::kFLOAT if the layer has no inputs.
The returned data type must have a format that is supported by the plugin.
See also
supportsFormat()
Warning
DataType:kBOOL and DataType::kUINT8 are not supported.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
9.108 nvinfer1::IPluginV2Ext Class Reference
545
9.108.3.8 getTensorRTVersion()
int32 t nvinfer1::IPluginV2Ext::getTensorRTVersion ( ) const [inline], [override], [protected],
[virtual], [noexcept]
Return the API version with which this plugin was built. The upper byte reserved by TensorRT and is used to differen-
tiate this from IPluginV2.
Do not override this method as it is used by the TensorRT library to maintain backwards-compatibility with plugins.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, the implementation provided here is safe to call from any thread.
Reimplemented from nvinfer1::IPluginV2.
Reimplemented in nvinfer1::IPluginV2IOExt.
9.108.3.9 isOutputBroadcastAcrossBatch()
virtual bool nvinfer1::IPluginV2Ext::isOutputBroadcastAcrossBatch (
int32 t outputIndex,
bool const ∗ inputIsBroadcasted,
int32 t nbInputs ) const [pure virtual], [noexcept]
Return true if output tensor is broadcast across a batch.
Parameters
outputIndex
The index of the output
inputIsBroadcasted
The ith element is true if the tensor for the ith input is broadcast across a batch.
nbInputs
The number of inputs
The values in inputIsBroadcasted refer to broadcasting at the semantic level, i.e. are unaffected by whether method
canBroadcastInputAcrossBatch requests physical replication of the values.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
546
Class Documentation
The documentation for this class was generated from the following file:
• NvInferRuntimePlugin.h
9.109
nvinfer1::IPluginV2IOExt Class Reference
Plugin class for user-implemented layers.
#include <NvInferRuntimePlugin.h>
Inheritance diagram for nvinfer1::IPluginV2IOExt:
nvinfer1::IPluginV2
nvinfer1::IPluginV2Ext
nvinfer1::IPluginV2IOExt
Public Member Functions
• virtual void configurePlugin (PluginTensorDesc const ∗in, int32 t nbInput, PluginTensorDesc const ∗out, int32←↩
t nbOutput) noexcept=0
Configure the layer.
• virtual bool supportsFormatCombination (int32 t pos, PluginTensorDesc const ∗inOut, int32 t nbInputs, int32 t
nbOutputs) const noexcept=0
Return true if plugin supports the format and datatype for the input/output indexed by pos.
Protected Member Functions
• int32 t getTensorRTVersion () const noexcept override
Return the API version with which this plugin was built. The upper byte is reserved by TensorRT and is used to differen-
tiate this from IPluginV2 and IPluginV2Ext.
9.109.1
Detailed Description
Plugin class for user-implemented layers.
Plugins are a mechanism for applications to implement custom layers. This interface provides additional capabilities
to the IPluginV2Ext interface by extending different I/O data types and tensor formats.
See also
IPluginV2Ext
9.109 nvinfer1::IPluginV2IOExt Class Reference
547
9.109.2
Member Function Documentation
9.109.2.1 configurePlugin()
virtual void nvinfer1::IPluginV2IOExt::configurePlugin (
PluginTensorDesc const ∗ in,
int32 t nbInput,
PluginTensorDesc const ∗ out,
int32 t nbOutput ) [pure virtual], [noexcept]
Configure the layer.
This function is called by the builder prior to initialize(). It provides an opportunity for the layer to make algorithm
choices on the basis of the provided I/O PluginTensorDesc.
Parameters
in
The input tensors attributes that are used for configuration.
nbInput
Number of input tensors.
out
The output tensors attributes that are used for configuration.
nbOutput
Number of output tensors.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin. However, TensorRT will not call this
method from two threads simultaneously on a given clone of a plugin.
9.109.2.2 getTensorRTVersion()
int32 t nvinfer1::IPluginV2IOExt::getTensorRTVersion ( ) const [inline], [override], [protected],
[virtual], [noexcept]
Return the API version with which this plugin was built. The upper byte is reserved by TensorRT and is used to
differentiate this from IPluginV2 and IPluginV2Ext.
Do not override this method as it is used by the TensorRT library to maintain backwards-compatibility with plugins.
Usage considerations
548
Class Documentation
• Allowed context for the API call
- Thread-safe: Yes, the implementation provided here is safe to call from any thread.
Reimplemented from nvinfer1::IPluginV2Ext.
9.109.2.3 supportsFormatCombination()
virtual bool nvinfer1::IPluginV2IOExt::supportsFormatCombination (
int32 t pos,
PluginTensorDesc const ∗ inOut,
int32 t nbInputs,
int32 t nbOutputs ) const [pure virtual], [noexcept]
Return true if plugin supports the format and datatype for the input/output indexed by pos.
For this method inputs are numbered 0..(nbInputs-1) and outputs are numbered nbInputs..(nbInputs+nbOutputs-1).
Using this numbering, pos is an index into InOut, where 0 <= pos < nbInputs+nbOutputs.
TensorRT invokes this method to ask if the input/output indexed by pos supports the format/datatype specified by
inOut[pos].format and inOut[pos].type. The override should return true if that format/datatype at inOut[pos] are sup-
ported by the plugin. If support is conditional on other input/output formats/datatypes, the plugin can make its result
conditional on the formats/datatypes in inOut[0..pos-1], which will be set to values that the plugin supports. The
override should not inspect inOut[pos+1..nbInputs+nbOutputs-1], which will have invalid values. In other words, the
decision for pos must be based on inOut[0..pos] only.
Some examples:
• A definition for a plugin that supports only FP16 NCHW:
return inOut.format[pos] == TensorFormat::kLINEAR && inOut.type[pos] == DataType::kHALF;
• A definition for a plugin that supports only FP16 NCHW for its two inputs, and FP32 NCHW for its single
output:
return inOut.format[pos] == TensorFormat::kLINEAR &&
(inOut.type[pos] == (pos < 2 ? DataType::kHALF : DataType::kFLOAT));
• A definition for a ”polymorphic” plugin with two inputs and one output that supports any format or type, but the
inputs and output must have the same format and type:
return pos == 0 || (inOut.format[pos] == inOut.format[0] && inOut.type[pos] == inOut.type[0]);
Warning: TensorRT will stop asking for formats once it finds kFORMAT COMBINATION LIMIT on combina-
tions.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
building networks on multiple devices sharing the same plugin.
The documentation for this class was generated from the following file:
• NvInferRuntimePlugin.h
9.110 nvinfer1::IPluginV2Layer Class Reference
549
9.110
nvinfer1::IPluginV2Layer Class Reference
Layer type for pluginV2.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IPluginV2Layer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IPluginV2Layer
Public Member Functions
• IPluginV2 & getPlugin () noexcept
Get the plugin for the layer.
Protected Member Functions
• virtual ∼IPluginV2Layer () noexcept=default
Protected Attributes
• apiv::VPluginV2Layer ∗ mImpl
9.110.1
Detailed Description
Layer type for pluginV2.
See also
IPluginV2
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.110.2
Constructor & Destructor Documentation
550
Class Documentation
9.110.2.1 ∼IPluginV2Layer()
virtual nvinfer1::IPluginV2Layer::∼IPluginV2Layer ( ) [protected], [virtual], [default], [noexcept]
9.110.3
Member Function Documentation
9.110.3.1 getPlugin()
IPluginV2 & nvinfer1::IPluginV2Layer::getPlugin ( ) [inline], [noexcept]
Get the plugin for the layer.
See also
IPluginV2
9.110.4
Member Data Documentation
9.110.4.1 mImpl
apiv::VPluginV2Layer∗ nvinfer1::IPluginV2Layer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.111
nvinfer1::IPoolingLayer Class Reference
A Pooling layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IPoolingLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IPoolingLayer
9.111 nvinfer1::IPoolingLayer Class Reference
551
Public Member Functions
•
void setPoolingType (PoolingType type) noexcept
Set the type of activation to be performed.
•
PoolingType getPoolingType () const noexcept
Get the type of activation to be performed.
•
TRT DEPRECATED void setWindowSize (DimsHW windowSize) noexcept
Set the window size for pooling.
•
TRT DEPRECATED DimsHW getWindowSize () const noexcept
Get the window size for pooling.
•
TRT DEPRECATED void setStride (DimsHW stride) noexcept
Set the stride for pooling.
•
TRT DEPRECATED DimsHW getStride () const noexcept
Get the stride for pooling.
•
TRT DEPRECATED void setPadding (DimsHW padding) noexcept
Set the padding for pooling.
•
TRT DEPRECATED DimsHW getPadding () const noexcept
Get the padding for pooling.
•
void setBlendFactor (float blendFactor) noexcept
Set the blending factor for the max average blend mode: max average blendPool = (1-blendFactor)∗maxPool + blend←↩
Factor∗avgPool blendFactor is a user value in [0,1] with the default value of 0.0 This value only applies for the kMAX←↩
AVERAGE BLEND mode.
•
float getBlendFactor () const noexcept
Get the blending factor for the max average blend mode: max average blendPool = (1-blendFactor)∗maxPool +
blendFactor∗avgPool blendFactor is a user value in [0,1] with the default value of 0.0 In modes other than kMAX←↩
AVERAGE BLEND, blendFactor is ignored.
•
void setAverageCountExcludesPadding (bool exclusive) noexcept
Set whether average pooling uses as a denominator the overlap area between the window and the unpadded input. If this
is not set, the denominator is the overlap between the pooling window and the padded input.
•
bool getAverageCountExcludesPadding () const noexcept
Get whether average pooling uses as a denominator the overlap area between the window and the unpadded input.
•
void setPrePadding (Dims padding) noexcept
Set the multi-dimension pre-padding for pooling.
•
Dims getPrePadding () const noexcept
Get the pre-padding.
•
void setPostPadding (Dims padding) noexcept
Set the multi-dimension post-padding for pooling.
•
Dims getPostPadding () const noexcept
Get the padding.
•
void setPaddingMode (PaddingMode paddingMode) noexcept
Set the padding mode.
•
PaddingMode getPaddingMode () const noexcept
Get the padding mode.
•
void setWindowSizeNd (Dims windowSize) noexcept
Set the multi-dimension window size for pooling.
•
Dims getWindowSizeNd () const noexcept
Get the multi-dimension window size for pooling.
•
void setStrideNd (Dims stride) noexcept
552
Class Documentation
Set the multi-dimension stride for pooling.
• Dims getStrideNd () const noexcept
Get the multi-dimension stride for pooling.
• void setPaddingNd (Dims padding) noexcept
Set the multi-dimension padding for pooling.
• Dims getPaddingNd () const noexcept
Get the multi-dimension padding for pooling.
Protected Member Functions
• virtual ∼IPoolingLayer () noexcept=default
Protected Attributes
• apiv::VPoolingLayer ∗ mImpl
9.111.1
Detailed Description
A Pooling layer in a network definition.
The layer applies a reduction operation within a window over the input.
Warning
When running pooling layer with DeviceType::kDLA in Int8 mode, the dynamic ranges for input and output
tensors must be equal.
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.111.2
Constructor & Destructor Documentation
9.111.2.1 ∼IPoolingLayer()
virtual nvinfer1::IPoolingLayer::∼IPoolingLayer ( ) [protected], [virtual], [default], [noexcept]
9.111.3
Member Function Documentation
9.111 nvinfer1::IPoolingLayer Class Reference
553
9.111.3.1 getAverageCountExcludesPadding()
bool nvinfer1::IPoolingLayer::getAverageCountExcludesPadding ( ) const [inline], [noexcept]
Get whether average pooling uses as a denominator the overlap area between the window and the unpadded input.
See also
setAverageCountExcludesPadding()
9.111.3.2 getBlendFactor()
float nvinfer1::IPoolingLayer::getBlendFactor ( ) const [inline], [noexcept]
Get the blending factor for the max average blend mode: max average blendPool = (1-blendFactor)∗maxPool +
blendFactor∗avgPool blendFactor is a user value in [0,1] with the default value of 0.0 In modes other than kMAX ←↩
AVERAGE BLEND, blendFactor is ignored.
See also
setBlendFactor()
9.111.3.3 getPadding()
TRT DEPRECATED DimsHW nvinfer1::IPoolingLayer::getPadding ( ) const [inline], [noexcept]
Get the padding for pooling.
Default: 0
See also
setPadding()
Deprecated Superseded by getPaddingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
554
Class Documentation
9.111.3.4 getPaddingMode()
PaddingMode nvinfer1::IPoolingLayer::getPaddingMode ( ) const [inline], [noexcept]
Get the padding mode.
Default: kEXPLICIT ROUND DOWN
See also
setPaddingMode()
9.111.3.5 getPaddingNd()
Dims nvinfer1::IPoolingLayer::getPaddingNd ( ) const [inline], [noexcept]
Get the multi-dimension padding for pooling.
If the padding is asymmetric, the pre-padding is returned.
See also
setPaddingNd()
9.111.3.6 getPoolingType()
PoolingType nvinfer1::IPoolingLayer::getPoolingType ( ) const [inline], [noexcept]
Get the type of activation to be performed.
See also
setPoolingType(), PoolingType
9.111.3.7 getPostPadding()
Dims nvinfer1::IPoolingLayer::getPostPadding ( ) const [inline], [noexcept]
Get the padding.
See also
setPostPadding()
9.111 nvinfer1::IPoolingLayer Class Reference
555
9.111.3.8 getPrePadding()
Dims nvinfer1::IPoolingLayer::getPrePadding ( ) const [inline], [noexcept]
Get the pre-padding.
See also
setPrePadding()
9.111.3.9 getStride()
TRT DEPRECATED DimsHW nvinfer1::IPoolingLayer::getStride ( ) const [inline], [noexcept]
Get the stride for pooling.
See also
setStride()
Deprecated Superseded by getStrideNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.111.3.10 getStrideNd()
Dims nvinfer1::IPoolingLayer::getStrideNd ( ) const [inline], [noexcept]
Get the multi-dimension stride for pooling.
See also
setStrideNd()
9.111.3.11 getWindowSize()
TRT DEPRECATED DimsHW nvinfer1::IPoolingLayer::getWindowSize ( ) const [inline], [noexcept]
Get the window size for pooling.
See also
setWindowSize()
Deprecated Superseded by getWindowSizeNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
556
Class Documentation
9.111.3.12 getWindowSizeNd()
Dims nvinfer1::IPoolingLayer::getWindowSizeNd ( ) const [inline], [noexcept]
Get the multi-dimension window size for pooling.
See also
setWindowSizeNd()
9.111.3.13 setAverageCountExcludesPadding()
void nvinfer1::IPoolingLayer::setAverageCountExcludesPadding (
bool exclusive ) [inline], [noexcept]
Set whether average pooling uses as a denominator the overlap area between the window and the unpadded input. If
this is not set, the denominator is the overlap between the pooling window and the padded input.
Default: true
Note
On Xavier, DLA supports only inclusive padding and this must be explicitly set to false.
See also
getAverageCountExcludesPadding()
9.111.3.14 setBlendFactor()
void nvinfer1::IPoolingLayer::setBlendFactor (
float blendFactor ) [inline], [noexcept]
Set the blending factor for the max average blend mode: max average blendPool = (1-blendFactor)∗maxPool +
blendFactor∗avgPool blendFactor is a user value in [0,1] with the default value of 0.0 This value only applies for
the kMAX AVERAGE BLEND mode.
Since DLA does not support kMAX AVERAGE BLEND, blendFactor is ignored on the DLA.
See also
getBlendFactor()
|
||
|
|
|