|
|
9.111 nvinfer1::IPoolingLayer Class Reference
557
9.111.3.15 setPadding()
TRT DEPRECATED void nvinfer1::IPoolingLayer::setPadding (
DimsHW padding ) [inline], [noexcept]
Set the padding for pooling.
Default: 0
If executing this layer on DLA, both height and width of padding must be in the range [0,7].
See also
getPadding()
Deprecated Superseded by setPaddingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.111.3.16 setPaddingMode()
void nvinfer1::IPoolingLayer::setPaddingMode (
PaddingMode paddingMode ) [inline], [noexcept]
Set the padding mode.
Padding mode takes precedence if both setPaddingMode and setPre/PostPadding are used.
Default: kEXPLICIT ROUND DOWN
See also
getPaddingMode()
9.111.3.17 setPaddingNd()
void nvinfer1::IPoolingLayer::setPaddingNd (
Dims padding ) [inline], [noexcept]
Set the multi-dimension padding for pooling.
The input will be padded by this number of elements in each dimension. Padding is symmetric. Padding value depends
on pooling type, -inf is used for max pooling and zero padding for average pooling.
Default: (0, 0, ..., 0)
If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range [0,7].
See also
getPaddingNd() setPadding() getPadding()
558
Class Documentation
9.111.3.18 setPoolingType()
void nvinfer1::IPoolingLayer::setPoolingType (
PoolingType type ) [inline], [noexcept]
Set the type of activation to be performed.
DLA only supports kMAX and kAVERAGE pooling types.
See also
getPoolingType(), PoolingType
9.111.3.19 setPostPadding()
void nvinfer1::IPoolingLayer::setPostPadding (
Dims padding ) [inline], [noexcept]
Set the multi-dimension post-padding for pooling.
The end of the input will be padded by this number of elements in each dimension. Padding value depends on pooling
type, -inf is used for max pooling and zero padding for average pooling.
Default: (0, 0, ..., 0)
If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range [0,7].
See also
getPostPadding()
9.111.3.20 setPrePadding()
void nvinfer1::IPoolingLayer::setPrePadding (
Dims padding ) [inline], [noexcept]
Set the multi-dimension pre-padding for pooling.
The start of the input will be padded by this number of elements in each dimension. Padding value depends on pooling
type, -inf is used for max pooling and zero padding for average pooling.
Default: (0, 0, ..., 0)
If executing this layer on DLA, only support 2D padding, both height and width of padding must be in the range [0,7].
See also
getPrePadding()
9.111 nvinfer1::IPoolingLayer Class Reference
559
9.111.3.21 setStride()
TRT DEPRECATED void nvinfer1::IPoolingLayer::setStride (
DimsHW stride ) [inline], [noexcept]
Set the stride for pooling.
Default: 1
If executing this layer on DLA, both height and width of stride must be in the range [1,16].
See also
getStride()
Deprecated Superseded by setStrideNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
9.111.3.22 setStrideNd()
void nvinfer1::IPoolingLayer::setStrideNd (
Dims stride ) [inline], [noexcept]
Set the multi-dimension stride for pooling.
Default: (1, 1, ..., 1)
If executing this layer on DLA, only support 2D stride, both height and width of stride must be in the range [1,16].
See also
getStrideNd() setStride() getStride()
9.111.3.23 setWindowSize()
TRT DEPRECATED void nvinfer1::IPoolingLayer::setWindowSize (
DimsHW windowSize ) [inline], [noexcept]
Set the window size for pooling.
If executing this layer on DLA, both height and width of window size must be in the range [1,8].
See also
getWindowSize()
Deprecated Superseded by setWindowSizeNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0
560
Class Documentation
9.111.3.24 setWindowSizeNd()
void nvinfer1::IPoolingLayer::setWindowSizeNd (
Dims windowSize ) [inline], [noexcept]
Set the multi-dimension window size for pooling.
If executing this layer on DLA, only support 2D window size, both height and width of window size must be in the
range [1,8].
See also
getWindowSizeNd() setWindowSize() getWindowSize()
9.111.4
Member Data Documentation
9.111.4.1 mImpl
apiv::VPoolingLayer∗ nvinfer1::IPoolingLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.112
nvinfer1::IProfiler Class Reference
Application-implemented interface for profiling.
#include <NvInferRuntime.h>
Public Member Functions
• virtual void reportLayerTime (char const ∗layerName, float ms) noexcept=0
Layer time reporting callback.
• virtual ∼IProfiler () noexcept
9.112.1
Detailed Description
Application-implemented interface for profiling.
When this class is added to an execution context, the profiler will be called once per layer for each invocation of
executeV2()/enqueueV2()/enqueueV3().
It is not recommended to run inference with profiler enabled when the inference execution time is critical since the
profiler may affect execution time negatively.
9.113 nvinfer1::IQuantizeLayer Class Reference
561
9.112.2
Constructor & Destructor Documentation
9.112.2.1 ∼IProfiler()
virtual nvinfer1::IProfiler::∼IProfiler ( ) [inline], [virtual], [noexcept]
9.112.3
Member Function Documentation
9.112.3.1 reportLayerTime()
virtual void nvinfer1::IProfiler::reportLayerTime (
char const ∗ layerName,
float ms ) [pure virtual], [noexcept]
Layer time reporting callback.
Parameters
layerName
The name of the layer, set when constructing the network definition. If the engine is built with
profiling verbosity set to kNONE, the layerName is the decimal index of the layer.
ms
The time in milliseconds to execute the layer.
The documentation for this class was generated from the following file:
• NvInferRuntime.h
9.113
nvinfer1::IQuantizeLayer Class Reference
A Quantize layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IQuantizeLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IQuantizeLayer
562
Class Documentation
Public Member Functions
• int32 t getAxis () const noexcept
Get the quantization axis.
• void setAxis (int32 t axis) noexcept
Set the quantization axis.
Protected Member Functions
• virtual ∼IQuantizeLayer () noexcept=default
Protected Attributes
• apiv::VQuantizeLayer ∗ mImpl
9.113.1
Detailed Description
A Quantize layer in a network definition.
This layer accepts a floating-point data input tensor, and uses the scale and zeroPt inputs to quantize the data to an 8-bit
signed integer according to: output = clamp(round(input / scale) + zeroPt)
Rounding type is rounding-to-nearest ties-to-even ( https://en.wikipedia.org/wiki/Rounding#←↩
Round half to even). Clamping is in the range [-128, 127].
The first input (index 0) is the tensor to be quantized. The second (index 1) and third (index 2) are the scale and zero
point respectively. Each of scale and zeroPt must be either a scalar, or a 1D tensor.
The zeroPt tensor is optional, and if not set, will be assumed to be zero. Its data type must be DataType::kINT8.
zeroPt must only contain zero-valued coefficients, because only symmetric quantization is supported. The scale
value must be either a scalar for per-tensor quantization, or a 1D tensor for per-channel quantization. All scale
coefficients must have positive values. The size of the 1-D scale tensor must match the size of the quantization axis.
The size of the scale must match the size of the zeroPt.
The subgraph which terminates with the scale tensor must be a build-time constant. The same restrictions apply to
the zeroPt. The output type, if constrained, must be constrained to DataType::kINT8. The input type, if constrained,
must be constrained to DataType::kFLOAT or DataType::kHALF. The output size is the same as the input size. The
quantization axis is in reference to the input tensor's dimensions.
IQuantizeLayer only supports DataType::kFLOAT precision and will default to this precision during instantiation.
IQuantizeLayer only supports DataType::kINT8 output.
As an example of the operation of this layer, imagine a 4D NCHW activation input which can be quantized using a
single scale coefficient (referred to as per-tensor quantization): For each n in N: For each c in C: For each h in H: For
each w in W: output[n,c,h,w] = clamp(round(input[n,c,h,w] / scale) + zeroPt)
Per-channel quantization is supported only for weight inputs. Thus, Activations cannot be quantized per-channel. As
an example of per-channel operation, imagine a 4D KCRS weights input and K (dimension 0) as the quantization axis.
The scale is an array of coefficients, and must have the same size as the quantization axis. For each k in K: For each c
in C: For each r in R: For each s in S: output[k,c,r,s] = clamp(round(input[k,c,r,s] / scale[k]) + zeroPt[k])
9.113 nvinfer1::IQuantizeLayer Class Reference
563
Note
Only symmetric quantization is supported.
Currently the only allowed build-time constant scale and zeroPt subgraphs are:
1. Constant -> Quantize
2. Constant -> Cast -> Quantize
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.113.2
Constructor & Destructor Documentation
9.113.2.1 ∼IQuantizeLayer()
virtual nvinfer1::IQuantizeLayer::∼IQuantizeLayer ( ) [protected], [virtual], [default], [noexcept]
9.113.3
Member Function Documentation
9.113.3.1 getAxis()
int32 t nvinfer1::IQuantizeLayer::getAxis ( ) const [inline], [noexcept]
Get the quantization axis.
Returns
axis parameter set by setAxis(). The return value is the index of the quantization axis in the input tensor's
dimensions. A value of -1 indicates per-tensor quantization. The default value is -1.
9.113.3.2 setAxis()
void nvinfer1::IQuantizeLayer::setAxis (
int32 t axis ) [inline], [noexcept]
Set the quantization axis.
Set the index of the quantization axis (with reference to the input tensor's dimensions). The axis must be a valid axis
if the scale tensor has more than one coefficient. The axis value will be ignored if the scale tensor has exactly one
coefficient (per-tensor quantization).
564
Class Documentation
9.113.4
Member Data Documentation
9.113.4.1 mImpl
apiv::VQuantizeLayer∗ nvinfer1::IQuantizeLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.114
nvinfer1::IRaggedSoftMaxLayer Class Reference
A RaggedSoftmax layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IRaggedSoftMaxLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IRaggedSoftMaxLayer
Protected Member Functions
• virtual ∼IRaggedSoftMaxLayer () noexcept=default
Protected Attributes
• apiv::VRaggedSoftMaxLayer ∗ mImpl
Additional Inherited Members
9.114.1
Detailed Description
A RaggedSoftmax layer in a network definition.
This layer takes a ZxS input tensor and an additional Zx1 bounds tensor holding the lengths of the Z sequences.
This layer computes a softmax across each of the Z sequences.
The output tensor is of the same size as the input tensor.
9.115 nvinfer1::IRecurrenceLayer Class Reference
565
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.114.2
Constructor & Destructor Documentation
9.114.2.1 ∼IRaggedSoftMaxLayer()
virtual nvinfer1::IRaggedSoftMaxLayer::∼IRaggedSoftMaxLayer ( ) [protected], [virtual], [default],
[noexcept]
9.114.3
Member Data Documentation
9.114.3.1 mImpl
apiv::VRaggedSoftMaxLayer∗ nvinfer1::IRaggedSoftMaxLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.115
nvinfer1::IRecurrenceLayer Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IRecurrenceLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::ILoopBoundaryLayer
nvinfer1::IRecurrenceLayer
566
Class Documentation
Public Member Functions
• void setInput (int32 t index, ITensor &tensor) noexcept
Append or replace an input of this layer with a specific tensor.
Protected Member Functions
• virtual ∼IRecurrenceLayer () noexcept=default
Protected Attributes
• apiv::VRecurrenceLayer ∗ mImpl
9.115.1
Constructor & Destructor Documentation
9.115.1.1 ∼IRecurrenceLayer()
virtual nvinfer1::IRecurrenceLayer::∼IRecurrenceLayer ( ) [protected], [virtual], [default],
[noexcept]
9.115.2
Member Function Documentation
9.115.2.1 setInput()
void nvinfer1::ILayer::setInput (
int32 t index,
ITensor & tensor ) [inline], [noexcept]
Append or replace an input of this layer with a specific tensor.
Parameters
index
the index of the input to modify.
tensor
the new input tensor Sets the input tensor for the given index.
For a recurrence layer, the values 0 and 1 are valid. The indices are as follows:
•
0: The initial value of the output tensor. The value must come from outside the loop.
9.116 nvinfer1::IReduceLayer Class Reference
567
•
1: The next value of the output tensor. The value usually comes from inside the loop, and must have the same
dimensions as input 0.
If this function is called with the value 1, then the function getNbInputs() changes from returning 1 to 2.
9.115.3
Member Data Documentation
9.115.3.1 mImpl
apiv::VRecurrenceLayer∗ nvinfer1::IRecurrenceLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.116
nvinfer1::IReduceLayer Class Reference
Layer that represents a reduction across a non-bool tensor.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IReduceLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IReduceLayer
Public Member Functions
• void setOperation (ReduceOperation op) noexcept
Set the reduce operation for the layer.
• ReduceOperation getOperation () const noexcept
Get the reduce operation for the layer.
• void setReduceAxes (uint32 t reduceAxes) noexcept
Set the axes over which to reduce.
• uint32 t getReduceAxes () const noexcept
Get the axes over which to reduce for the layer.
• void setKeepDimensions (bool keepDimensions) noexcept
Set the boolean that specifies whether or not to keep the reduced dimensions for the layer.
• bool getKeepDimensions () const noexcept
Get the boolean that specifies whether or not to keep the reduced dimensions for the layer.
568
Class Documentation
Protected Member Functions
• virtual ∼IReduceLayer () noexcept=default
Protected Attributes
• apiv::VReduceLayer ∗ mImpl
9.116.1
Detailed Description
Layer that represents a reduction across a non-bool tensor.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.116.2
Constructor & Destructor Documentation
9.116.2.1 ∼IReduceLayer()
virtual nvinfer1::IReduceLayer::∼IReduceLayer ( ) [protected], [virtual], [default], [noexcept]
9.116.3
Member Function Documentation
9.116.3.1 getKeepDimensions()
bool nvinfer1::IReduceLayer::getKeepDimensions ( ) const [inline], [noexcept]
Get the boolean that specifies whether or not to keep the reduced dimensions for the layer.
See also
setKeepDimensions
9.116 nvinfer1::IReduceLayer Class Reference
569
9.116.3.2 getOperation()
ReduceOperation nvinfer1::IReduceLayer::getOperation ( ) const [inline], [noexcept]
Get the reduce operation for the layer.
See also
setOperation(), ReduceOperation
9.116.3.3 getReduceAxes()
uint32 t nvinfer1::IReduceLayer::getReduceAxes ( ) const [inline], [noexcept]
Get the axes over which to reduce for the layer.
See also
setReduceAxes
9.116.3.4 setKeepDimensions()
void nvinfer1::IReduceLayer::setKeepDimensions (
bool keepDimensions ) [inline], [noexcept]
Set the boolean that specifies whether or not to keep the reduced dimensions for the layer.
See also
getKeepDimensions
9.116.3.5 setOperation()
void nvinfer1::IReduceLayer::setOperation (
ReduceOperation op ) [inline], [noexcept]
Set the reduce operation for the layer.
See also
getOperation(), ReduceOperation
570
Class Documentation
9.116.3.6 setReduceAxes()
void nvinfer1::IReduceLayer::setReduceAxes (
uint32 t reduceAxes ) [inline], [noexcept]
Set the axes over which to reduce.
See also
getReduceAxes
9.116.4
Member Data Documentation
9.116.4.1 mImpl
apiv::VReduceLayer∗ nvinfer1::IReduceLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.117
nvinfer1::IRefitter Class Reference
Updates weights in an engine.
#include <NvInferRuntime.h>
Inheritance diagram for nvinfer1::IRefitter:
nvinfer1::INoCopy
nvinfer1::IRefitter
9.117 nvinfer1::IRefitter Class Reference
571
Public Member Functions
•
virtual ∼IRefitter () noexcept=default
•
bool setWeights (char const ∗layerName, WeightsRole role, Weights weights) noexcept
Specify new weights for a layer of given name. Returns true on success, or false if new weights are rejected. Possible
reasons for rejection are:
•
bool refitCudaEngine () noexcept
Updates associated engine. Return true if successful.
•
int32 t getMissing (int32 t size, char const ∗∗layerNames, WeightsRole ∗roles) noexcept
Get description of missing weights.
•
int32 t getAll (int32 t size, char const ∗∗layerNames, WeightsRole ∗roles) noexcept
Get description of all weights that could be refit.
•
TRT DEPRECATED void destroy () noexcept
•
bool setDynamicRange (char const ∗tensorName, float min, float max) noexcept
•
float getDynamicRangeMin (char const ∗tensorName) const noexcept
Get minimum of dynamic range.
•
float getDynamicRangeMax (char const ∗tensorName) const noexcept
Get maximum of dynamic range.
•
int32 t getTensorsWithDynamicRange (int32 t size, char const ∗∗tensorNames) const noexcept
Get names of all tensors that have refittable dynamic ranges.
•
void setErrorRecorder (IErrorRecorder ∗recorder) noexcept
Set the ErrorRecorder for this interface.
•
IErrorRecorder ∗ getErrorRecorder () const noexcept
Get the ErrorRecorder assigned to this interface.
•
bool setNamedWeights (char const ∗name, Weights weights) noexcept
Specify new weights of given name.
•
int32 t getMissingWeights (int32 t size, char const ∗∗weightsNames) noexcept
Get names of missing weights.
•
int32 t getAllWeights (int32 t size, char const ∗∗weightsNames) noexcept
Get names of all weights that could be refit.
•
ILogger ∗ getLogger () const noexcept
get the logger with which the refitter was created
•
bool setMaxThreads (int32 t maxThreads) noexcept
Set the maximum number of threads.
•
int32 t getMaxThreads () const noexcept
get the maximum number of threads that can be used by the refitter.
Protected Attributes
• apiv::VRefitter ∗ mImpl
Additional Inherited Members
9.117.1
Detailed Description
Updates weights in an engine.
572
Class Documentation
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.117.2
Constructor & Destructor Documentation
9.117.2.1 ∼IRefitter()
virtual nvinfer1::IRefitter::∼IRefitter ( ) [virtual], [default], [noexcept]
9.117.3
Member Function Documentation
9.117.3.1 destroy()
TRT DEPRECATED void nvinfer1::IRefitter::destroy ( ) [inline], [noexcept]
Deprecated Deprecated in TRT 8.0. Superseded by delete.
Warning
Calling destroy on a managed pointer will result in a double-free error.
9.117.3.2 getAll()
int32 t nvinfer1::IRefitter::getAll (
int32 t size,
char const ∗∗ layerNames,
WeightsRole ∗ roles ) [inline], [noexcept]
Get description of all weights that could be refit.
Parameters
size
The number of items that can be safely written to a non-null layerNames or roles.
layerNames
Where to write the layer names.
roles
Where to write the weights roles.
9.117 nvinfer1::IRefitter Class Reference
573
Returns
The number of Weights that could be refit.
If layerNames!=nullptr, each written pointer points to a string owned by the engine being refit, and becomes invalid
when the engine is destroyed.
9.117.3.3 getAllWeights()
int32 t nvinfer1::IRefitter::getAllWeights (
int32 t size,
char const ∗∗ weightsNames ) [inline], [noexcept]
Get names of all weights that could be refit.
Parameters
size
The number of weights names that can be safely written to.
weightsNames
The names of the weights to be updated, or nullptr for unnamed weights.
Returns
The number of Weights that could be refit.
If layerNames!=nullptr, each written pointer points to a string owned by the engine being refit, and becomes invalid
when the engine is destroyed.
9.117.3.4 getDynamicRangeMax()
float nvinfer1::IRefitter::getDynamicRangeMax (
char const ∗ tensorName ) const [inline], [noexcept]
Get maximum of dynamic range.
Returns
Maximum of dynamic range.
If the dynamic range was never set, returns the maximum computed during calibration.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
574
Class Documentation
9.117.3.5 getDynamicRangeMin()
float nvinfer1::IRefitter::getDynamicRangeMin (
char const ∗ tensorName ) const [inline], [noexcept]
Get minimum of dynamic range.
Returns
Minimum of dynamic range.
If the dynamic range was never set, returns the minimum computed during calibration.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.117.3.6 getErrorRecorder()
IErrorRecorder ∗ nvinfer1::IRefitter::getErrorRecorder ( ) const [inline], [noexcept]
Get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A nullptr will be returned if an error handler has not
been set.
Returns
A pointer to the IErrorRecorder object that has been registered.
See also
setErrorRecorder()
9.117.3.7 getLogger()
ILogger ∗ nvinfer1::IRefitter::getLogger ( ) const [inline], [noexcept]
get the logger with which the refitter was created
Returns
the logger
9.117 nvinfer1::IRefitter Class Reference
575
9.117.3.8 getMaxThreads()
int32 t nvinfer1::IRefitter::getMaxThreads ( ) const [inline], [noexcept]
get the maximum number of threads that can be used by the refitter.
Retrieves the maximum number of threads that can be used by the refitter.
Returns
The maximum number of threads that can be used by the refitter.
See also
setMaxThreads()
9.117.3.9 getMissing()
int32 t nvinfer1::IRefitter::getMissing (
int32 t size,
char const ∗∗ layerNames,
WeightsRole ∗ roles ) [inline], [noexcept]
Get description of missing weights.
For example, if some Weights have been set, but the engine was optimized in a way that combines weights, any
unsupplied Weights in the combination are considered missing.
Parameters
size
The number of items that can be safely written to a non-null layerNames or roles.
layerNames
Where to write the layer names.
roles
Where to write the weights roles.
Returns
The number of missing Weights.
If layerNames!=nullptr, each written pointer points to a string owned by the engine being refit, and becomes invalid
when the engine is destroyed.
9.117.3.10 getMissingWeights()
int32 t nvinfer1::IRefitter::getMissingWeights (
int32 t size,
char const ∗∗ weightsNames ) [inline], [noexcept]
576
Class Documentation
Get names of missing weights.
For example, if some Weights have been set, but the engine was optimized in a way that combines weights, any
unsupplied Weights in the combination are considered missing.
Parameters
size
The number of weights names that can be safely written to.
weightsNames
The names of the weights to be updated, or nullptr for unnamed weights.
Returns
The number of missing Weights.
If layerNames!=nullptr, each written pointer points to a string owned by the engine being refit, and becomes invalid
when the engine is destroyed.
9.117.3.11 getTensorsWithDynamicRange()
int32 t nvinfer1::IRefitter::getTensorsWithDynamicRange (
int32 t size,
char const ∗∗ tensorNames ) const [inline], [noexcept]
Get names of all tensors that have refittable dynamic ranges.
Parameters
size
The number of items that can be safely written to a non-null tensorNames.
tensorNames
Where to write the layer names.
Returns
The number of Weights that could be refit.
If tensorNames!=nullptr, each written pointer points to a string owned by the engine being refit, and becomes invalid
when the engine is destroyed.
9.117.3.12 refitCudaEngine()
bool nvinfer1::IRefitter::refitCudaEngine ( ) [inline], [noexcept]
Updates associated engine. Return true if successful.
Failure occurs if getMissing() != 0 before the call.
The behavior is undefined if the engine has pending enqueued work.
Extant IExecutionContexts associated with the engine should not be used afterwards. Instead, create new IExecution←↩
Contexts after refitting.
9.117 nvinfer1::IRefitter Class Reference
577
9.117.3.13 setDynamicRange()
bool nvinfer1::IRefitter::setDynamicRange (
char const ∗ tensorName,
float min,
float max ) [inline], [noexcept]
Update dynamic range for a tensor.
Parameters
tensorName
The name of an ITensor in the network.
min
The minimum of the dynamic range for the tensor.
max
The maximum of the dynamic range for the tensor.
Returns
True if successful; false otherwise.
Returns false if there is no Int8 engine tensor derived from a network tensor of that name. If successful, then getMissing
may report that some weights need to be supplied.
Warning
The string tensorName must be null-terminated, and be at most 4096 bytes including the terminator.
9.117.3.14 setErrorRecorder()
void nvinfer1::IRefitter::setErrorRecorder (
IErrorRecorder ∗ recorder ) [inline], [noexcept]
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function
will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder
with the interface, resulting in a call to decRefCount if a recorder has been registered.
If an error recorder is not set, messages will be sent to the global log stream.
Parameters
recorder
The error recorder to register with this interface.
578
Class Documentation
See also
getErrorRecorder()
9.117.3.15 setMaxThreads()
bool nvinfer1::IRefitter::setMaxThreads (
int32 t maxThreads ) [inline], [noexcept]
Set the maximum number of threads.
Parameters
maxThreads
The maximum number of threads that can be used by the refitter.
Returns
True if successful, false otherwise.
The default value is 1 and includes the current thread. A value greater than 1 permits TensorRT to use multi-threaded
algorithms. A value less than 1 triggers a kINVALID ARGUMENT error.
9.117.3.16 setNamedWeights()
bool nvinfer1::IRefitter::setNamedWeights (
char const ∗ name,
Weights weights ) [inline], [noexcept]
Specify new weights of given name.
Parameters
name
The name of the weights to be refit.
weights
The new weights to associate with the name.
Returns true on success, or false if new weights are rejected. Possible reasons for rejection are:
• The name of weights is nullptr or does not correspond to any refittable weights.
• The number of weights is inconsistent with the original specification.
Modifying the weights before method refitCudaEngine() completes will result in undefined behavior.
9.117 nvinfer1::IRefitter Class Reference
579
Warning
The string name must be null-terminated, and be at most 4096 bytes including the terminator.
9.117.3.17 setWeights()
bool nvinfer1::IRefitter::setWeights (
char const ∗ layerName,
WeightsRole role,
Weights weights ) [inline], [noexcept]
Specify new weights for a layer of given name. Returns true on success, or false if new weights are rejected. Possible
reasons for rejection are:
• There is no such layer by that name.
• The layer does not have weights with the specified role.
• The number of weights is inconsistent with the layer’s original specification.
Modifying the weights before method refit() completes will result in undefined behavior.
Warning
The string layerName must be null-terminated, and be at most 4096 bytes including the terminator.
9.117.4
Member Data Documentation
9.117.4.1 mImpl
apiv::VRefitter∗ nvinfer1::IRefitter::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInferRuntime.h
580
Class Documentation
9.118
nvinfer1::IResizeLayer Class Reference
A resize layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IResizeLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IResizeLayer
Public Member Functions
•
void setOutputDimensions (Dims dimensions) noexcept
Set the output dimensions.
•
Dims getOutputDimensions () const noexcept
Get the output dimensions.
•
void setScales (float const ∗scales, int32 t nbScales) noexcept
Set the resize scales.
•
int32 t getScales (int32 t size, float ∗scales) const noexcept
Copies resize scales to scales[0, ..., nbScales-1], where nbScales is the number of scales that were set.
•
void setResizeMode (ResizeMode resizeMode) noexcept
Set resize mode for an input tensor.
•
ResizeMode getResizeMode () const noexcept
Get resize mode for an input tensor.
•
TRT DEPRECATED void setAlignCorners (bool alignCorners) noexcept
Set whether to align corners while resizing.
•
TRT DEPRECATED bool getAlignCorners () const noexcept
True if align corners has been set.
•
void setCoordinateTransformation (ResizeCoordinateTransformation coordTransform) noexcept
Set coordinate transformation function.
•
ResizeCoordinateTransformation getCoordinateTransformation () const noexcept
Get coordinate transformation function.
•
void setSelectorForSinglePixel (ResizeSelector selector) noexcept
Set coordinate selector function when resized to single pixel.
•
ResizeSelector getSelectorForSinglePixel () const noexcept
Get the coordinate selector function when resized to single pixel.
•
void setNearestRounding (ResizeRoundMode value) noexcept
Set rounding mode for nearest neighbor resize.
•
ResizeRoundMode getNearestRounding () const noexcept
Get rounding mode for nearest neighbor resize.
•
void setCubicCoeff (float A) noexcept
9.118 nvinfer1::IResizeLayer Class Reference
581
Set the coefficient 'A' used in cubic interpolation.
• float getCubicCoeff () const noexcept
Get the coefficient 'A' used in cubic interpolation.
• void setExcludeOutside (bool excludeFlag) noexcept
Set the state for excluding outside pixels.
• bool getExcludeOutside () const noexcept
Get the state for excluding outside pixels.
• void setInput (int32 t index, ITensor &tensor) noexcept
Append or replace an input of this layer with a specific tensor.
Protected Member Functions
• virtual ∼IResizeLayer () noexcept=default
Protected Attributes
• apiv::VResizeLayer ∗ mImpl
9.118.1
Detailed Description
A resize layer in a network definition.
Resize layer can be used for resizing a N-D tensor.
Resize layer currently supports the following configurations:
• ResizeMode::kNEAREST - resizes innermost m dimensions of N-D, where 0 < m <= min(8, N) and N > 0
• ResizeMode::kLINEAR - resizes innermost m dimensions of N-D, where 0 < m <= min(3, N) and N > 0
Default resize mode is ResizeMode::kNEAREST.
The coordinates in the output tensor are mapped to coordinates in the input tensor using a function set by call-
ing setCoordinateTransformation(). The default for all ResizeMode settings (nearest, linear, bilinear, etc.) is
ResizeCoordinateTransformation::kASYMMETRIC.
The resize layer provides two ways to resize tensor dimensions.
• Set output dimensions directly. It can be done for static as well as dynamic resize layer. Static resize layer
requires output dimensions to be known at build-time. Dynamic resize layer requires output dimensions to be
set as one of the input tensors.
• Set scales for resize. Each output dimension is calculated as floor(input dimension ∗ scale). Only static resize
layer allows setting scales where the scales are known at build-time.
If executing this layer on DLA, the following combinations of parameters are supported:
582
Class Documentation
• In kNEAREST mode:
- (ResizeCoordinateTransformation::kASYMMETRIC, ResizeSelector::kFORMULA, ResizeRoundMode::kFLOOR)
- (ResizeCoordinateTransformation::kHALF PIXEL, ResizeSelector::kFORMULA, ResizeRoundMode::kHALF DOWN)
- (ResizeCoordinateTransformation::kHALF PIXEL, ResizeSelector::kFORMULA, ResizeRoundMode::kHALF UP)
• In kLINEAR mode:
- (ResizeCoordinateTransformation::kHALF PIXEL, ResizeSelector::kFORMULA)
- (ResizeCoordinateTransformation::kHALF PIXEL, ResizeSelector::kUPPER)
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.118.2
Constructor & Destructor Documentation
9.118.2.1 ∼IResizeLayer()
virtual nvinfer1::IResizeLayer::∼IResizeLayer ( ) [protected], [virtual], [default], [noexcept]
9.118.3
Member Function Documentation
9.118.3.1 getAlignCorners()
TRT DEPRECATED bool nvinfer1::IResizeLayer::getAlignCorners ( ) const [inline], [noexcept]
True if align corners has been set.
Returns
True if align corners has been set, false otherwise.
Deprecated Deprecated in TensorRT 8.0. Superseded by IResizeLayer::getCoordinateTransformation().
9.118 nvinfer1::IResizeLayer Class Reference
583
9.118.3.2 getCoordinateTransformation()
ResizeCoordinateTransformation nvinfer1::IResizeLayer::getCoordinateTransformation ( ) const [inline],
[noexcept]
Get coordinate transformation function.
Returns
The coordinate transformation function.
9.118.3.3 getCubicCoeff()
float nvinfer1::IResizeLayer::getCubicCoeff ( ) const [inline], [noexcept]
Get the coefficient 'A' used in cubic interpolation.
See also
setCubicCoeff()
9.118.3.4 getExcludeOutside()
bool nvinfer1::IResizeLayer::getExcludeOutside ( ) const [inline], [noexcept]
Get the state for excluding outside pixels.
See also
setExcludeOutside()
9.118.3.5 getNearestRounding()
ResizeRoundMode nvinfer1::IResizeLayer::getNearestRounding ( ) const [inline], [noexcept]
Get rounding mode for nearest neighbor resize.
Returns
The rounding mode.
584
Class Documentation
9.118.3.6 getOutputDimensions()
Dims nvinfer1::IResizeLayer::getOutputDimensions ( ) const [inline], [noexcept]
Get the output dimensions.
Returns
The output dimensions.
9.118.3.7 getResizeMode()
ResizeMode nvinfer1::IResizeLayer::getResizeMode ( ) const [inline], [noexcept]
Get resize mode for an input tensor.
Returns
The resize mode.
9.118.3.8 getScales()
int32 t nvinfer1::IResizeLayer::getScales (
int32 t size,
float ∗ scales ) const [inline], [noexcept]
Copies resize scales to scales[0, ..., nbScales-1], where nbScales is the number of scales that were set.
Parameters
size
The number of scales to get. If size != nbScales, no scales will be copied.
scales
Pointer to where to copy the scales. Scales will be copied only if size == nbScales and scales != nullptr.
In case the size is not known consider using size = 0 and scales = nullptr. This method will return the number of resize
scales.
Returns
The number of resize scales i.e. nbScales if scales were set. Return -1 in case no scales were set or resize layer is
used in dynamic mode.
9.118 nvinfer1::IResizeLayer Class Reference
585
9.118.3.9 getSelectorForSinglePixel()
ResizeSelector nvinfer1::IResizeLayer::getSelectorForSinglePixel ( ) const [inline], [noexcept]
Get the coordinate selector function when resized to single pixel.
Returns
The selector function.
9.118.3.10 setAlignCorners()
TRT DEPRECATED void nvinfer1::IResizeLayer::setAlignCorners (
bool alignCorners ) [inline], [noexcept]
Set whether to align corners while resizing.
If true, the centers of the 4 corner pixels of both input and output tensors are aligned i.e. preserves the values of corner
pixels.
Default: false.
Deprecated Deprecated in TensorRT 8.0. Superseded by IResizeLayer::setCoordinateTransformation().
9.118.3.11 setCoordinateTransformation()
void nvinfer1::IResizeLayer::setCoordinateTransformation (
ResizeCoordinateTransformation coordTransform ) [inline], [noexcept]
Set coordinate transformation function.
The function maps a coordinate in the output tensor to a coordinate in the input tensor.
Default function is ResizeCoordinateTransformation::kASYMMETRIC.
See also
ResizeCoordinateTransformation
586
Class Documentation
9.118.3.12 setCubicCoeff()
void nvinfer1::IResizeLayer::setCubicCoeff (
float A ) [inline], [noexcept]
Set the coefficient 'A' used in cubic interpolation.
Cubic uses the coefficient 'A' to calculate the weight of input pixels:
x := The relative distance between the sampled pixels and the input coordinates.
weight(x) := for |x| <= 1, ((A + 2) * x - (A + 3)) * x * x + 1,
for 1 < |x| < 2, ((A * x - 5 * A) * x + 8 * A) * x - 4 * A,
others 0;
This attribute is valid only if ”resize mode” is ”cubic”.
The default value is -0.75.
9.118.3.13 setExcludeOutside()
void nvinfer1::IResizeLayer::setExcludeOutside (
bool excludeFlag ) [inline], [noexcept]
Set the state for excluding outside pixels.
If set to true, the weight of sampling locations outside the input tensor will be set to false, and the weight will be
renormalized so that their sum is 1.0.
The default value is false.
9.118.3.14 setInput()
void nvinfer1::ILayer::setInput (
int32 t index,
ITensor & tensor ) [inline], [noexcept]
Append or replace an input of this layer with a specific tensor.
Parameters
index
the index of the input to modify.
tensor
the new input tensor.
Sets the input tensor for the given index. The index must be 0 for a static resize layer. A static resize layer is converted
to a dynamic resize layer by calling setInput with an index 1. A dynamic resize layer cannot be converted back to a
static resize layer.
9.118 nvinfer1::IResizeLayer Class Reference
587
For a dynamic resize layer, the values 0 and 1 are valid. The indices in the dynamic case are as follows:
• 0: Execution tensor to be resized.
• 1: The output dimensions, as a 1D Int32 shape tensor.
If this function is called with the value 1, then the function getNbInputs() changes from returning 1 to 2.
9.118.3.15 setNearestRounding()
void nvinfer1::IResizeLayer::setNearestRounding (
ResizeRoundMode value ) [inline], [noexcept]
Set rounding mode for nearest neighbor resize.
This value is used for nearest neighbor interpolation rounding. It is applied after coordinate transformation.
Default is kFLOOR.
See also
ResizeRoundMode
9.118.3.16 setOutputDimensions()
void nvinfer1::IResizeLayer::setOutputDimensions (
Dims dimensions ) [inline], [noexcept]
Set the output dimensions.
Parameters
dimensions
The output dimensions. Number of output dimensions must be the same as the number of input
dimensions.
If executing this layer on DLA, setOutputDimensions() is not supported.
If there is a second input, i.e. resize layer is dynamic, calling setOutputDimensions() is an error and does not update
the dimensions.
Output dimensions can be specified directly, or via scale factors relative to input dimensions. Scales for resize can be
provided using setScales().
588
Class Documentation
See also
setScales
getOutputDimensions
9.118.3.17 setResizeMode()
void nvinfer1::IResizeLayer::setResizeMode (
ResizeMode resizeMode ) [inline], [noexcept]
Set resize mode for an input tensor.
Supported resize modes are Nearest Neighbor and Linear.
See also
ResizeMode
9.118.3.18 setScales()
void nvinfer1::IResizeLayer::setScales (
float const ∗ scales,
int32 t nbScales ) [inline], [noexcept]
Set the resize scales.
Parameters
scales
An array of resize scales.
nbScales
Number of scales. Number of scales must be equal to the number of input dimensions.
If executing this layer on DLA, there are three restrictions: 1) nbScales has to be exactly 4. 2) the first two elements
in scales need to be exactly 1 (for unchanged batch and channel dimensions). 3) The last two elements in scales,
representing the scale values along height and width dimensions, respectively, need to be integer values in the range of
[1, 32] for kNEAREST mode and [1, 4] for kLINEAR. Example of DLA-supported scales: {1, 1, 2, 2}.
If there is a second input, i.e. resize layer is dynamic, calling setScales() is an error and does not update the scales.
Output dimensions are calculated as follows: outputDims[i] = floor(inputDims[i] ∗ scales[i])
Output dimensions can be specified directly, or via scale factors relative to input dimensions. Output dimensions can
be provided directly using setOutputDimensions().
9.119 nvinfer1::IReverseSequenceLayer Class Reference
589
See also
setOutputDimensions
getScales
9.118.3.19 setSelectorForSinglePixel()
void nvinfer1::IResizeLayer::setSelectorForSinglePixel (
ResizeSelector selector ) [inline], [noexcept]
Set coordinate selector function when resized to single pixel.
When resize to single pixel image, use this function to decide how to map the coordinate in the original image.
Default is ResizeSelector::kFORMULA.
See also
ResizeSelector
9.118.4
Member Data Documentation
9.118.4.1 mImpl
apiv::VResizeLayer∗ nvinfer1::IResizeLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.119
nvinfer1::IReverseSequenceLayer Class Reference
A ReverseSequence layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IReverseSequenceLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IReverseSequenceLayer
590
Class Documentation
Public Member Functions
• void setBatchAxis (int32 t batchAxis) noexcept
Set the batch axis. Default is 1.
• int32 t getBatchAxis () const noexcept
Return the batch axis. Return 1 if no batch axis was set.
• void setSequenceAxis (int32 t sequenceAxis) noexcept
Set the sequence axis. Default is 0.
• int32 t getSequenceAxis () const noexcept
Return the sequence axis. Return 0 if no sequence axis was set.
Protected Member Functions
• virtual ∼IReverseSequenceLayer () noexcept=default
Protected Attributes
• apiv::VReverseSequenceLayer ∗ mImpl
9.119.1
Detailed Description
A ReverseSequence layer in a network definition.
This layer performs batch-wise reversal, which slices the input tensor along the axis batchAxis. For the i-th slice, the
operation reverses the first N elements, specified by the corresponding i-th value in sequenceLens, along sequenceAxis
and keeps the remaining elements unchanged. The output tensor will have the same shape as the input tensor.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.119.2
Constructor & Destructor Documentation
9.119.2.1 ∼IReverseSequenceLayer()
virtual nvinfer1::IReverseSequenceLayer::∼IReverseSequenceLayer ( ) [protected], [virtual], [default],
[noexcept]
9.119.3
Member Function Documentation
9.119 nvinfer1::IReverseSequenceLayer Class Reference
591
9.119.3.1 getBatchAxis()
int32 t nvinfer1::IReverseSequenceLayer::getBatchAxis ( ) const [inline], [noexcept]
Return the batch axis. Return 1 if no batch axis was set.
See also
getBatchAxis()
9.119.3.2 getSequenceAxis()
int32 t nvinfer1::IReverseSequenceLayer::getSequenceAxis ( ) const [inline], [noexcept]
Return the sequence axis. Return 0 if no sequence axis was set.
See also
getSequenceAxis()
9.119.3.3 setBatchAxis()
void nvinfer1::IReverseSequenceLayer::setBatchAxis (
int32 t batchAxis ) [inline], [noexcept]
Set the batch axis. Default is 1.
batchAxis should be between zero (inclusive) and the rank of input (exclusive), and different from sequenceAxis.
Otherwise, ErrorCode::kINVALID ARGUMENT will be triggered.
See also
setBatchAxis()
9.119.3.4 setSequenceAxis()
void nvinfer1::IReverseSequenceLayer::setSequenceAxis (
int32 t sequenceAxis ) [inline], [noexcept]
Set the sequence axis. Default is 0.
sequenceAxis should be between zero (inclusive) and the rank of input (exclusive), and different from batchAxis.
Otherwise, ErrorCode::kINVALID ARGUMENT will be triggered.
See also
setSequenceAxis()
592
Class Documentation
9.119.4
Member Data Documentation
9.119.4.1 mImpl
apiv::VReverseSequenceLayer∗ nvinfer1::IReverseSequenceLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.120
nvinfer1::IRNNv2Layer Class Reference
An RNN layer in a network definition, version 2.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IRNNv2Layer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IRNNv2Layer
Public Member Functions
• int32 t getLayerCount () const noexcept
Get the layer count of the RNN.
• int32 t getHiddenSize () const noexcept
Get the hidden size of the RNN.
• int32 t getMaxSeqLength () const noexcept
Get the maximum sequence length of the RNN.
• int32 t getDataLength () const noexcept
Get the embedding length of the RNN.
• void setSequenceLengths (ITensor &seqLengths) noexcept
Specify individual sequence lengths in the batch with the ITensor pointed to by seqLengths.
• ITensor ∗ getSequenceLengths () const noexcept
Get the sequence lengths specified for the RNN.
• void setOperation (RNNOperation op) noexcept
Set the operation of the RNN layer.
9.120 nvinfer1::IRNNv2Layer Class Reference
593
• RNNOperation getOperation () const noexcept
Get the operation of the RNN layer.
• void setInputMode (RNNInputMode op) noexcept
Set the input mode of the RNN layer.
• RNNInputMode getInputMode () const noexcept
Get the input mode of the RNN layer.
• void setDirection (RNNDirection op) noexcept
Set the direction of the RNN layer.
• RNNDirection getDirection () const noexcept
Get the direction of the RNN layer.
• void setWeightsForGate (int32 t layerIndex, RNNGateType gate, bool isW, Weights weights) noexcept
Set the weight parameters for an individual gate in the RNN.
• Weights getWeightsForGate (int32 t layerIndex, RNNGateType gate, bool isW) const noexcept
Get the weight parameters for an individual gate in the RNN.
• void setBiasForGate (int32 t layerIndex, RNNGateType gate, bool isW, Weights bias) noexcept
Set the bias parameters for an individual gate in the RNN.
• Weights getBiasForGate (int32 t layerIndex, RNNGateType gate, bool isW) const noexcept
Get the bias parameters for an individual gate in the RNN.
• void setHiddenState (ITensor &hidden) noexcept
Set the initial hidden state of the RNN with the provided hidden ITensor.
• ITensor ∗ getHiddenState () const noexcept
Get the initial hidden state of the RNN.
• void setCellState (ITensor &cell) noexcept
Set the initial cell state of the LSTM with the provided cell ITensor.
• ITensor ∗ getCellState () const noexcept
Get the initial cell state of the RNN.
Protected Member Functions
• virtual ∼IRNNv2Layer () noexcept=default
Protected Attributes
• apiv::VRNNv2Layer ∗ mImpl
9.120.1
Detailed Description
An RNN layer in a network definition, version 2.
This layer supersedes IRNNLayer.
Deprecated Deprecated prior to TensorRT 8.0 and will be removed in 9.0. Superseded by INetworkDefinition::addLoop().
594
Class Documentation
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.120.2
Constructor & Destructor Documentation
9.120.2.1 ∼IRNNv2Layer()
virtual nvinfer1::IRNNv2Layer::∼IRNNv2Layer ( ) [protected], [virtual], [default], [noexcept]
9.120.3
Member Function Documentation
9.120.3.1 getBiasForGate()
Weights nvinfer1::IRNNv2Layer::getBiasForGate (
int32 t layerIndex,
RNNGateType gate,
bool isW ) const [inline], [noexcept]
Get the bias parameters for an individual gate in the RNN.
See also
setBiasForGate()
9.120.3.2 getCellState()
ITensor ∗ nvinfer1::IRNNv2Layer::getCellState ( ) const [inline], [noexcept]
Get the initial cell state of the RNN.
See also
setCellState()
9.120 nvinfer1::IRNNv2Layer Class Reference
595
9.120.3.3 getDataLength()
int32 t nvinfer1::IRNNv2Layer::getDataLength ( ) const [inline], [noexcept]
Get the embedding length of the RNN.
9.120.3.4 getDirection()
RNNDirection nvinfer1::IRNNv2Layer::getDirection ( ) const [inline], [noexcept]
Get the direction of the RNN layer.
See also
setDirection(), RNNDirection
9.120.3.5 getHiddenSize()
int32 t nvinfer1::IRNNv2Layer::getHiddenSize ( ) const [inline], [noexcept]
Get the hidden size of the RNN.
9.120.3.6 getHiddenState()
ITensor ∗ nvinfer1::IRNNv2Layer::getHiddenState ( ) const [inline], [noexcept]
Get the initial hidden state of the RNN.
See also
setHiddenState()
596
Class Documentation
9.120.3.7 getInputMode()
RNNInputMode nvinfer1::IRNNv2Layer::getInputMode ( ) const [inline], [noexcept]
Get the input mode of the RNN layer.
See also
setInputMode(), RNNInputMode
9.120.3.8 getLayerCount()
int32 t nvinfer1::IRNNv2Layer::getLayerCount ( ) const [inline], [noexcept]
Get the layer count of the RNN.
9.120.3.9 getMaxSeqLength()
int32 t nvinfer1::IRNNv2Layer::getMaxSeqLength ( ) const [inline], [noexcept]
Get the maximum sequence length of the RNN.
9.120.3.10 getOperation()
RNNOperation nvinfer1::IRNNv2Layer::getOperation ( ) const [inline], [noexcept]
Get the operation of the RNN layer.
See also
setOperation(), RNNOperation
9.120 nvinfer1::IRNNv2Layer Class Reference
597
9.120.3.11 getSequenceLengths()
ITensor ∗ nvinfer1::IRNNv2Layer::getSequenceLengths ( ) const [inline], [noexcept]
Get the sequence lengths specified for the RNN.
Returns
nullptr if no sequence lengths were specified, the sequence length data otherwise.
See also
setSequenceLengths()
9.120.3.12 getWeightsForGate()
Weights nvinfer1::IRNNv2Layer::getWeightsForGate (
int32 t layerIndex,
RNNGateType gate,
bool isW ) const [inline], [noexcept]
Get the weight parameters for an individual gate in the RNN.
See also
setWeightsForGate()
9.120.3.13 setBiasForGate()
void nvinfer1::IRNNv2Layer::setBiasForGate (
int32 t layerIndex,
RNNGateType gate,
bool isW,
Weights bias ) [inline], [noexcept]
Set the bias parameters for an individual gate in the RNN.
The DataType for this structure must be DataType::kFLOAT or DataType::kHALF, and must be the same datatype as
the input tensor.
Each bias vector has a fixed size, getHiddenSize().
598
Class Documentation
Parameters
layerIndex
The index of the layer that contains this gate. See setWeightsForGate() for a description of the layer
index.
gate
The name of the gate within the RNN layer. The gate name must correspond to one of the gates used
by this layer's RNNOperation.
isW
True if the bias parameters are for the input bias Wb[g] and false if they are for the recurrent input
bias Rb[g]. See RNNOperation for equations showing how these bias vectors are used in the RNN
gate.
bias
The weight structure holding the bias parameters, which should be an array of size getHiddenSize().
9.120.3.14 setCellState()
void nvinfer1::IRNNv2Layer::setCellState (
ITensor & cell ) [inline], [noexcept]
Set the initial cell state of the LSTM with the provided cell ITensor.
The cell ITensor should have the dimensions {N1, ..., Np, L, H}, where:
• N1..Np are the index dimensions specified by the input tensor
• ListhenumberoflayersintheRNN,equaltogetLayerCount()ifgetDirectionisRNNDirection::kUNIDIRECTION,
and 2x getLayerCount() if getDirection is RNNDirection::kBIDIRECTION. In the bi-directional case, layer l's
final forward hidden state is stored in L = 2∗l, and final backward hidden state is stored in L= 2∗l + 1.
• H is the hidden state for each layer, equal to getHiddenSize().
It is an error to call setCellState() on an RNN layer that is not configured with RNNOperation::kLSTM.
9.120.3.15 setDirection()
void nvinfer1::IRNNv2Layer::setDirection (
RNNDirection op ) [inline], [noexcept]
Set the direction of the RNN layer.
The direction determines if the RNN is run as a unidirectional(left to right) or bidirectional(left to right and right to
left). In the RNNDirection::kBIDIRECTION case the output is concatenated together, resulting in output size of 2x
getHiddenSize().
See also
getDirection(), RNNDirection
9.120 nvinfer1::IRNNv2Layer Class Reference
599
9.120.3.16 setHiddenState()
void nvinfer1::IRNNv2Layer::setHiddenState (
ITensor & hidden ) [inline], [noexcept]
Set the initial hidden state of the RNN with the provided hidden ITensor.
The hidden ITensor should have the dimensions {N1, ..., Np, L, H}, where:
• N1..Np are the index dimensions specified by the input tensor
• ListhenumberoflayersintheRNN,equaltogetLayerCount()ifgetDirectionisRNNDirection::kUNIDIRECTION,
and 2x getLayerCount() if getDirection is RNNDirection::kBIDIRECTION. In the bi-directional case, layer l's
final forward hidden state is stored in L = 2∗l, and final backward hidden state is stored in L= 2∗l + 1.
• H is the hidden state for each layer, equal to getHiddenSize().
9.120.3.17 setInputMode()
void nvinfer1::IRNNv2Layer::setInputMode (
RNNInputMode op ) [inline], [noexcept]
Set the input mode of the RNN layer.
See also
getInputMode(), RNNInputMode
9.120.3.18 setOperation()
void nvinfer1::IRNNv2Layer::setOperation (
RNNOperation op ) [inline], [noexcept]
Set the operation of the RNN layer.
See also
getOperation(), RNNOperation
600
Class Documentation
9.120.3.19 setSequenceLengths()
void nvinfer1::IRNNv2Layer::setSequenceLengths (
ITensor & seqLengths ) [inline], [noexcept]
Specify individual sequence lengths in the batch with the ITensor pointed to by seqLengths.
The seqLengths ITensor should be a {N1, ..., Np} tensor, where N1..Np are the index dimensions of the input tensor
to the RNN.
If this is not specified, then the RNN layer assumes all sequences are size getMaxSeqLength().
All sequence lengths in seqLengths should be in the range [1, getMaxSeqLength()]. Zero-length sequences are not
supported.
This tensor must be of type DataType::kINT32.
9.120.3.20 setWeightsForGate()
void nvinfer1::IRNNv2Layer::setWeightsForGate (
int32 t layerIndex,
RNNGateType gate,
bool isW,
Weights weights ) [inline], [noexcept]
Set the weight parameters for an individual gate in the RNN.
The DataType for this structure must be DataType::kFLOAT or DataType::kHALF, and must be the same datatype as
the input tensor.
Each parameter matrix is row-major in memory, and has the following dimensions:
Let K := { ::kUNIDIRECTION => 1
{ ::kBIDIRECTION => 2
l := layer index (as described above)
H := getHiddenSize()
E := getDataLength() (the embedding length)
isW := true if the matrix is an input (W) matrix, and false if
the matrix is a recurrent input (R) matrix.
if isW:
if l < K and ::kSKIP:
(numRows, numCols) := (0, 0) # input matrix is skipped
elif l < K and ::kLINEAR:
(numRows, numCols) := (H, E) # input matrix acts on input data size E
elif l >= K:
(numRows, numCols) := (H, K * H) # input matrix acts on previous hidden state
else: # not isW
(numRows, numCols) := (H, H)
In other words, the input weights of the first layer of the RNN (if not skipped) transform a getDataLength()-size
column vector into a getHiddenSize()-size column vector. The input weights of subsequent layers transform a
K∗getHiddenSize()-size column vector into a getHiddenSize()-size column vector. K=2 in the bidirectional
case to account for the full hidden state being the concatenation of the forward and backward RNN hidden states.
The recurrent weight matrices for all layers all have shape (H, H), both in the unidirectional and bidirectional cases.
(In the bidirectional case, each recurrent weight matrix for the (forward or backward) RNN cell operates on the previous
(forward or backward) RNN cell's hidden state, which is size H).
9.121 nvinfer1::IRuntime Class Reference
601
Parameters
layerIndex
The index of the layer that contains this gate.
gate
The name of the gate within the RNN layer. The gate name must correspond to one of the gates used
by this layer's RNNOperation.
isW
True if the weight parameters are for the input matrix W[g] and false if they are for the recurrent
input matrix R[g]. See RNNOperation for equations showing how these matrices are used in the
RNN gate.
weights
The weight structure holding the weight parameters, which are stored as a row-major 2D matrix. See
See setWeightsForGate() for documentation on the expected dimensions of this matrix.
9.120.4
Member Data Documentation
9.120.4.1 mImpl
apiv::VRNNv2Layer∗ nvinfer1::IRNNv2Layer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.121
nvinfer1::IRuntime Class Reference
Allows a serialized functionally unsafe engine to be deserialized.
#include <NvInferRuntime.h>
Inheritance diagram for nvinfer1::IRuntime:
nvinfer1::INoCopy
nvinfer1::IRuntime
602
Class Documentation
Public Member Functions
•
virtual ∼IRuntime () noexcept=default
•
TRT DEPRECATED nvinfer1::ICudaEngine ∗ deserializeCudaEngine (void const ∗blob, std::size t size,
IPluginFactory ∗pluginFactory) noexcept
Deserialize an engine from a stream.
•
void setDLACore (int32 t dlaCore) noexcept
Sets the DLA core used by the network. Defaults to -1.
•
int32 t getDLACore () const noexcept
Get the DLA core that the engine executes on.
•
int32 t getNbDLACores () const noexcept
Returns number of DLA hardware cores accessible or 0 if DLA is unavailable.
•
TRT DEPRECATED void destroy () noexcept
Destroy this object.
•
void setGpuAllocator (IGpuAllocator ∗allocator) noexcept
Set the GPU allocator.
•
void setErrorRecorder (IErrorRecorder ∗recorder) noexcept
Set the ErrorRecorder for this interface.
•
IErrorRecorder ∗ getErrorRecorder () const noexcept
get the ErrorRecorder assigned to this interface.
•
ICudaEngine ∗ deserializeCudaEngine (void const ∗blob, std::size t size) noexcept
Deserialize an engine from a stream.
•
ILogger ∗ getLogger () const noexcept
get the logger with which the runtime was created
•
bool setMaxThreads (int32 t maxThreads) noexcept
Set the maximum number of threads.
•
int32 t getMaxThreads () const noexcept
Get the maximum number of threads that can be used by the runtime.
•
void setTemporaryDirectory (char const ∗path) noexcept
Set the directory that will be used by this runtime for temporary files.
•
char const ∗ getTemporaryDirectory () const noexcept
Get the directory that will be used by this runtime for temporary files.
•
void setTempfileControlFlags (TempfileControlFlags flags) noexcept
Set the tempfile control flags for this runtime.
•
TempfileControlFlags getTempfileControlFlags () const noexcept
Get the tempfile control flags for this runtime.
•
IPluginRegistry & getPluginRegistry () noexcept
Get the local plugin registry that can be used by the runtime.
•
IRuntime ∗ loadRuntime (char const ∗path) noexcept
Load IRuntime from the file.
•
void setEngineHostCodeAllowed (bool allowed) noexcept
Set whether the runtime is allowed to deserialize engines with host executable code.
•
bool getEngineHostCodeAllowed () const noexcept
Get whether the runtime is allowed to deserialize engines with host executable code.
Protected Attributes
• apiv::VRuntime ∗ mImpl
9.121 nvinfer1::IRuntime Class Reference
603
Additional Inherited Members
9.121.1
Detailed Description
Allows a serialized functionally unsafe engine to be deserialized.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.121.2
Constructor & Destructor Documentation
9.121.2.1 ∼IRuntime()
virtual nvinfer1::IRuntime::∼IRuntime ( ) [virtual], [default], [noexcept]
9.121.3
Member Function Documentation
9.121.3.1 deserializeCudaEngine() [1/2]
ICudaEngine ∗ nvinfer1::IRuntime::deserializeCudaEngine (
void const ∗ blob,
std::size t size ) [inline], [noexcept]
Deserialize an engine from a stream.
If an error recorder has been set for the runtime, it will also be passed to the engine.
Parameters
blob
The memory that holds the serialized engine.
size
The size of the memory.
Returns
The engine, or nullptr if it could not be deserialized.
604
Class Documentation
9.121.3.2 deserializeCudaEngine() [2/2]
TRT DEPRECATED nvinfer1::ICudaEngine ∗ nvinfer1::IRuntime::deserializeCudaEngine (
void const ∗ blob,
std::size t size,
IPluginFactory ∗ pluginFactory ) [inline], [noexcept]
Deserialize an engine from a stream.
If an error recorder has been set for the runtime, it will also be passed to the engine.
Parameters
blob
The memory that holds the serialized engine.
size
The size of the memory in bytes.
pluginFactory
The plugin factory, if any plugins are used by the network, otherwise nullptr.
Returns
The engine, or nullptr if it could not be deserialized.
Deprecated Deprecated in TensorRT 8.0.
Warning
IPluginFactory is no longer supported, therefore pluginFactory must be a nullptr.
9.121.3.3 destroy()
TRT DEPRECATED void nvinfer1::IRuntime::destroy ( ) [inline], [noexcept]
Destroy this object.
Deprecated Deprecated in TRT 8.0. Superseded by delete.
Warning
Calling destroy on a managed pointer will result in a double-free error.
9.121 nvinfer1::IRuntime Class Reference
605
9.121.3.4 getDLACore()
int32 t nvinfer1::IRuntime::getDLACore ( ) const [inline], [noexcept]
Get the DLA core that the engine executes on.
Returns
assigned DLA core or -1 for DLA not present or unset.
9.121.3.5 getEngineHostCodeAllowed()
bool nvinfer1::IRuntime::getEngineHostCodeAllowed ( ) const [inline], [noexcept]
Get whether the runtime is allowed to deserialize engines with host executable code.
Returns
Whether the runtime is allowed to deserialize engines with host executable code.
9.121.3.6 getErrorRecorder()
IErrorRecorder ∗ nvinfer1::IRuntime::getErrorRecorder ( ) const [inline], [noexcept]
get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A nullptr will be returned if an error handler has not
been set.
Returns
A pointer to the IErrorRecorder object that has been registered.
See also
setErrorRecorder()
606
Class Documentation
9.121.3.7 getLogger()
ILogger ∗ nvinfer1::IRuntime::getLogger ( ) const [inline], [noexcept]
get the logger with which the runtime was created
Returns
the logger
9.121.3.8 getMaxThreads()
int32 t nvinfer1::IRuntime::getMaxThreads ( ) const [inline], [noexcept]
Get the maximum number of threads that can be used by the runtime.
Retrieves the maximum number of threads that can be used by the runtime.
Returns
The maximum number of threads that can be used by the runtime.
See also
setMaxThreads()
9.121.3.9 getNbDLACores()
int32 t nvinfer1::IRuntime::getNbDLACores ( ) const [inline], [noexcept]
Returns number of DLA hardware cores accessible or 0 if DLA is unavailable.
9.121.3.10 getPluginRegistry()
IPluginRegistry & nvinfer1::IRuntime::getPluginRegistry ( ) [inline], [noexcept]
Get the local plugin registry that can be used by the runtime.
Returns
The local plugin registry that can be used by the runtime.
9.121 nvinfer1::IRuntime Class Reference
607
9.121.3.11 getTempfileControlFlags()
TempfileControlFlags nvinfer1::IRuntime::getTempfileControlFlags ( ) const [inline], [noexcept]
Get the tempfile control flags for this runtime.
Returns
The flags currently set.
See also
TempfileControlFlag, TempfileControlFlags, setTempfileControlFlags()
9.121.3.12 getTemporaryDirectory()
char const ∗ nvinfer1::IRuntime::getTemporaryDirectory ( ) const [inline], [noexcept]
Get the directory that will be used by this runtime for temporary files.
Returns
A path to the temporary directory in use, or nullptr if no path is specified.
See also
setTemporaryDirectory()
9.121.3.13 loadRuntime()
IRuntime ∗ nvinfer1::IRuntime::loadRuntime (
char const ∗ path ) [inline], [noexcept]
Load IRuntime from the file.
This method loads a runtime library from a shared library file. The runtime can then be used to execute a plan file built
with BuilderFlag::kVERSION COMPATIBLE and BuilderFlag::kEXCLUDE LEAN RUNTIME both set and built
with the same version of TensorRT as the loaded runtime library.
Parameters
path
Path to the runtime lean library.
608
Class Documentation
Returns
the runtime library, or nullptr if it could not be loaded
Warning
The path string must be null-terminated, and be at most 4096 bytes including the terminator.
9.121.3.14 setDLACore()
void nvinfer1::IRuntime::setDLACore (
int32 t dlaCore ) [inline], [noexcept]
Sets the DLA core used by the network. Defaults to -1.
Parameters
dlaCore
The DLA core to execute the engine on, in the range [0,getNbDlaCores()).
This function is used to specify which DLA core to use via indexing, if multiple DLA cores are available.
Warning
if getNbDLACores() returns 0, then this function does nothing.
See also
getDLACore()
9.121.3.15 setEngineHostCodeAllowed()
void nvinfer1::IRuntime::setEngineHostCodeAllowed (
bool allowed ) [inline], [noexcept]
Set whether the runtime is allowed to deserialize engines with host executable code.
Parameters
allowed
Whether the runtime is allowed to deserialize engines with host executable code.
The default value is false.
9.121 nvinfer1::IRuntime Class Reference
609
9.121.3.16 setErrorRecorder()
void nvinfer1::IRuntime::setErrorRecorder (
IErrorRecorder ∗ recorder ) [inline], [noexcept]
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function
will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder
with the interface, resulting in a call to decRefCount if a recorder has been registered.
If an error recorder is not set, messages will be sent to the global log stream.
Parameters
recorder
The error recorder to register with this interface.
See also
getErrorRecorder()
9.121.3.17 setGpuAllocator()
void nvinfer1::IRuntime::setGpuAllocator (
IGpuAllocator ∗ allocator ) [inline], [noexcept]
Set the GPU allocator.
Parameters
allocator
Set the GPU allocator to be used by the runtime. All GPU memory acquired will use this allocator. If
NULL is passed, the default allocator will be used.
Default: uses cudaMalloc/cudaFree.
If nullptr is passed, the default allocator will be used.
9.121.3.18 setMaxThreads()
bool nvinfer1::IRuntime::setMaxThreads (
int32 t maxThreads ) [inline], [noexcept]
Set the maximum number of threads.
610
Class Documentation
Parameters
maxThreads
The maximum number of threads that can be used by the runtime.
Returns
True if successful, false otherwise.
The default value is 1 and includes the current thread. A value greater than 1 permits TensorRT to use multi-threaded
algorithms. A value less than 1 triggers a kINVALID ARGUMENT error.
9.121.3.19 setTempfileControlFlags()
void nvinfer1::IRuntime::setTempfileControlFlags (
TempfileControlFlags flags ) [inline], [noexcept]
Set the tempfile control flags for this runtime.
Parameters
flags
The flags to set.
The default value is all flags set, i.e.
(1U << static cast<uint32 t>(kALLOW IN MEMORY FILES)) | (1U << static cast<uint32 t>(kALLOW ←↩
TEMPORARY FILES))
See also
TempfileControlFlag, TempfileControlFlags, getTempfileControlFlags()
9.121.3.20 setTemporaryDirectory()
void nvinfer1::IRuntime::setTemporaryDirectory (
char const ∗ path ) [inline], [noexcept]
Set the directory that will be used by this runtime for temporary files.
On some platforms the TensorRT runtime may need to create and use temporary files with read/write/execute permis-
sions to implement runtime functionality.
Parameters
path
Path to the temporary directory for use, or nullptr.
9.122 nvinfer1::safe::IRuntime Class Reference
611
If path is nullptr, then TensorRT will use platform-specific heuristics to pick a default temporary directory if required:
• On UNIX/Linux platforms, TensorRT will first try the TMPDIR environment variable, then fall back to /tmp
• On Windows, TensorRT will try the TEMP environment variable.
See the TensorRT Developer Guide for more information.
The default value is nullptr.
Warning
If path is not nullptr, it must be a non-empty string representing a relative or absolute path in the format
expected by the host operating system.
The string path must be null-terminated, and be at most 4096 bytes including the terminator. Note that the
operating system may have stricter path length requirements.
The process using TensorRT must have rwx permissions for the temporary directory, and the directory shall
be configured to disallow other users from modifying created files (e.g. on Linux, if the directory is shared
with other users, the sticky bit must be set).
See also
getTemporaryDirectory()
9.121.4
Member Data Documentation
9.121.4.1 mImpl
apiv::VRuntime∗ nvinfer1::IRuntime::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInferRuntime.h
9.122
nvinfer1::safe::IRuntime Class Reference
Allows a serialized functionally safe engine to be deserialized.
#include <NvInferSafeRuntime.h>
612
Class Documentation
Public Member Functions
• virtual ICudaEngine ∗ deserializeCudaEngine (void const ∗const blob, std::size t const size) noexcept=0
Deserialize an engine from a stream.
• virtual void setGpuAllocator (IGpuAllocator ∗const allocator) noexcept=0
Set the GPU allocator.
• virtual void setErrorRecorder (IErrorRecorder ∗const recorder) noexcept=0
Set the ErrorRecorder for this interface.
• virtual IErrorRecorder ∗ getErrorRecorder () const noexcept=0
Get the ErrorRecorder assigned to this interface.
• IRuntime ()=default
• virtual ∼IRuntime () noexcept=default
• IRuntime (IRuntime const &)=delete
• IRuntime (IRuntime &&)=delete
• IRuntime & operator= (IRuntime const &) &=delete
• IRuntime & operator= (IRuntime &&) &=delete
9.122.1
Detailed Description
Allows a serialized functionally safe engine to be deserialized.
Warning
In the safety runtime the application is required to set the error reporter for correct error handling.
See also
setErrorRecorder()
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.122.2
Constructor & Destructor Documentation
9.122.2.1 IRuntime() [1/3]
nvinfer1::safe::IRuntime::IRuntime ( ) [default]
9.122 nvinfer1::safe::IRuntime Class Reference
613
9.122.2.2 ∼IRuntime()
virtual nvinfer1::safe::IRuntime::∼IRuntime ( ) [virtual], [default], [noexcept]
9.122.2.3 IRuntime() [2/3]
nvinfer1::safe::IRuntime::IRuntime (
IRuntime const &
) [delete]
9.122.2.4 IRuntime() [3/3]
nvinfer1::safe::IRuntime::IRuntime (
IRuntime &&
) [delete]
9.122.3
Member Function Documentation
9.122.3.1 deserializeCudaEngine()
virtual ICudaEngine ∗ nvinfer1::safe::IRuntime::deserializeCudaEngine (
void const ∗const blob,
std::size t const size ) [pure virtual], [noexcept]
Deserialize an engine from a stream.
If the serialized engine requires plugins the plugin creator must be registered by calling IPluginRegistry::registerCreator()
before calling deserializeCudaEngine(). Every plugin creator registered must have a unique combination of namespace,
plugin name, and version.
Parameters
blob
The memory that holds the serialized engine.
size
The size of the memory in bytes.
Returns
The engine, or nullptr if it could not be deserialized.
614
Class Documentation
See also
IPluginRegistry::registerCreator()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, if called from different instances of safe::IRuntime. Calling deserializeCudaEngine of
the same safety runtime from multiple threads is not guaranteed to be thread safe.
9.122.3.2 getErrorRecorder()
virtual IErrorRecorder ∗ nvinfer1::safe::IRuntime::getErrorRecorder ( ) const [pure virtual],
[noexcept]
Get the ErrorRecorder assigned to this interface.
Retrieves the assigned error recorder object for the given class. A default error recorder does not exist, so a nullptr will
be returned if setErrorRecorder has not been called.
Returns
A pointer to the IErrorRecorder object that has been registered.
See also
setErrorRecorder()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.122.3.3 operator=() [1/2]
IRuntime & nvinfer1::safe::IRuntime::operator= (
IRuntime &&
) & [delete]
9.122 nvinfer1::safe::IRuntime Class Reference
615
9.122.3.4 operator=() [2/2]
IRuntime & nvinfer1::safe::IRuntime::operator= (
IRuntime const &
) & [delete]
9.122.3.5 setErrorRecorder()
virtual void nvinfer1::safe::IRuntime::setErrorRecorder (
IErrorRecorder ∗const recorder ) [pure virtual], [noexcept]
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function
will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder
with the interface, resulting in a call to decRefCount if a recorder has been registered.
Parameters
recorder
The error recorder to register with this interface.
See also
getErrorRecorder()
Usage considerations
• Allowed context for the API call
- Thread-safe: No
9.122.3.6 setGpuAllocator()
virtual void nvinfer1::safe::IRuntime::setGpuAllocator (
IGpuAllocator ∗const allocator ) [pure virtual], [noexcept]
Set the GPU allocator.
Parameters
allocator
Set the GPU allocator to be used by the runtime. All GPU memory acquired will use this allocator. If
NULL is passed, the default allocator will be used.
616
Class Documentation
Default: uses cudaMalloc/cudaFree.
If nullptr is passed, the default allocator will be used.
Usage considerations
• Allowed context for the API call
- Thread-safe: No
The documentation for this class was generated from the following file:
• NvInferSafeRuntime.h
9.123
nvinfer1::IScaleLayer Class Reference
A Scale layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IScaleLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IScaleLayer
Public Member Functions
• void setMode (ScaleMode mode) noexcept
Set the scale mode.
• ScaleMode getMode () const noexcept
Get the scale mode.
• void setShift (Weights shift) noexcept
Set the shift value.
• Weights getShift () const noexcept
Get the shift value.
• void setScale (Weights scale) noexcept
Set the scale value.
• Weights getScale () const noexcept
Get the scale value.
• void setPower (Weights power) noexcept
Set the power value.
• Weights getPower () const noexcept
Get the power value.
• int32 t getChannelAxis () const noexcept
Get the channel axis.
• void setChannelAxis (int32 t channelAxis) noexcept
Set the channel axis.
9.123 nvinfer1::IScaleLayer Class Reference
617
Protected Member Functions
• virtual ∼IScaleLayer () noexcept=default
Protected Attributes
• apiv::VScaleLayer ∗ mImpl
9.123.1
Detailed Description
A Scale layer in a network definition.
This layer applies a per-element computation to its input:
output = (input∗ scale + shift)∧ power
The coefficients can be applied on a per-tensor, per-channel, or per-element basis.
Note
If the number of weights is 0, then a default value is used for shift, power, and scale. The default shift is 0, the
default power is 1, and the default scale is 1.
The output size is the same as the input size.
Note
The input tensor for this layer is required to have a minimum of 3 dimensions in implicit batch mode and a
minimum of 4 dimensions in explicit batch mode.
A scale layer may be used as an INT8 quantization node in a graph, if the output is constrained to INT8 and the input
to FP32. Quantization rounds ties to even, and clamps to [-128, 127].
See also
ScaleMode
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.123.2
Constructor & Destructor Documentation
618
Class Documentation
9.123.2.1 ∼IScaleLayer()
virtual nvinfer1::IScaleLayer::∼IScaleLayer ( ) [protected], [virtual], [default], [noexcept]
9.123.3
Member Function Documentation
9.123.3.1 getChannelAxis()
int32 t nvinfer1::IScaleLayer::getChannelAxis ( ) const [inline], [noexcept]
Get the channel axis.
Returns
channelAxis parameter passed to addScaleNd() or set by setChannelAxis()
The value is the index of the channel axis in the input tensor's dimensions. Scaling happens along the channel axis
when ScaleMode::kCHANNEL is enabled.
See also
addScaleNd()
9.123.3.2 getMode()
ScaleMode nvinfer1::IScaleLayer::getMode ( ) const [inline], [noexcept]
Get the scale mode.
See also
setMode()
9.123.3.3 getPower()
Weights nvinfer1::IScaleLayer::getPower ( ) const [inline], [noexcept]
Get the power value.
See also
setPower()
9.123 nvinfer1::IScaleLayer Class Reference
619
9.123.3.4 getScale()
Weights nvinfer1::IScaleLayer::getScale ( ) const [inline], [noexcept]
Get the scale value.
See also
setScale()
9.123.3.5 getShift()
Weights nvinfer1::IScaleLayer::getShift ( ) const [inline], [noexcept]
Get the shift value.
See also
setShift()
9.123.3.6 setChannelAxis()
void nvinfer1::IScaleLayer::setChannelAxis (
int32 t channelAxis ) [inline], [noexcept]
Set the channel axis.
The value is the index of the channel axis in the input tensor's dimensions.
For ScaleMode::kCHANNEL, there can be distinct scale, shift, and power weights for each channel coordinate. For
ScaleMode::kELEMENTWISE, there can be distinct scale, shift, and power weights for each combination of coordi-
nates from the channel axis and axes after it.
For example, suppose the input tensor has dimensions [10,20,30,40] and the channel axis is 1. Let [n,c,h,w] de-
note an input coordinate. For ScaleMode::kCHANNEL, the scale, shift, and power weights are indexed by c. For
ScaleMode::kELEMENTWISE, the scale, shift, and power weights are indexed by [c,h,w].
See also
addScaleNd()
620
Class Documentation
9.123.3.7 setMode()
void nvinfer1::IScaleLayer::setMode (
ScaleMode mode ) [inline], [noexcept]
Set the scale mode.
See also
getMode()
9.123.3.8 setPower()
void nvinfer1::IScaleLayer::setPower (
Weights power ) [inline], [noexcept]
Set the power value.
See also
getPower()
9.123.3.9 setScale()
void nvinfer1::IScaleLayer::setScale (
Weights scale ) [inline], [noexcept]
Set the scale value.
See also
getScale()
9.123.3.10 setShift()
void nvinfer1::IScaleLayer::setShift (
Weights shift ) [inline], [noexcept]
Set the shift value.
See also
getShift()
9.124 nvinfer1::IScatterLayer Class Reference
621
9.123.4
Member Data Documentation
9.123.4.1 mImpl
apiv::VScaleLayer∗ nvinfer1::IScaleLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.124
nvinfer1::IScatterLayer Class Reference
A scatter layer in a network definition. Supports several kinds of scattering.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IScatterLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IScatterLayer
Public Member Functions
• void setMode (ScatterMode mode) noexcept
Set the scatter mode.
• ScatterMode getMode () const noexcept
Get the scatter mode.
• void setAxis (int32 t axis) noexcept
Set the axis used by ScatterMode::kELEMENTS.
• int32 t getAxis () const noexcept
Get the axis.
Protected Member Functions
• virtual ∼IScatterLayer () noexcept=default
622
Class Documentation
Protected Attributes
• apiv::VScatterLayer ∗ mImpl
9.124.1
Detailed Description
A scatter layer in a network definition. Supports several kinds of scattering.
The Scatter layer has three input tensors: Data, Indices, and Updates, one output tensor Output, and a scatter mode.
When kELEMENT mode is used an optional axis parameter is available.
• Data is a tensor of rank r >= 1 that stores the values to be duplicated in Output.
• Indices is a tensor of rank q that determines which locations in Output to write new values to. Constraints on the
rank of q depend on the mode: ScatterMode::kND: q >= 1 ScatterMode::kELEMENT: q must be the same as r
• Updates is atensor of rank s >=1 that provides the data to write to Output specified by its corresponding location
in Index. Constraints the rank of Updates depend on the mode: ScatterMode::kND: s = r + q - shape(Indices)[-1]
- 1 Scattermode::kELEMENT: s = q = r
• Output is a tensor with the same dimensions as Data that stores the resulting values of the transformation. It
must not be a shape tensor. The types of Data, Update, and Output shall be the same, and Indices shall be
DataType::kINT32.
The output is computed by copying the data, and then updating elements of it based on indices. How Indices are
interpreted depends upon the ScatterMode.
ScatterMode::kND
The indices are interpreted as a tensor of rank q-1 of indexing tuples.
The axis parameter is ignored.
Given that data dims are {d_0,...,d_{r-1}} and indices dims are {i_0,...,i_{q-1}},
define k = indices[q-1], it follows that updates dims are {i_0,...,i_{q-2},d_k,...,d_{r-1}}
The updating can be computed by:
foreach slice in indices[i_0,...i_{q-2}]
output[indices[slice]] = updates[slice]
ScatterMode::kELEMENT
Here "axis" denotes the result of getAxis().
For each element X of indices:
Let J denote a sequence for the subscripts of X
Let K = sequence J with element [axis] replaced by X
output[K] = updates[J]
For example, if indices has dimensions [N,C,H,W] and axis is 2, then the updates happen as:
for n in [0,n)
for c in [0,n)
for h in [0,n)
for w in [0,n)
output[n,c,indices[n,c,h,w],w] = updates[n,c,h,w]]
Writes to the same output element cause undefined behavior.
9.124 nvinfer1::IScatterLayer Class Reference
623
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.124.2
Constructor & Destructor Documentation
9.124.2.1 ∼IScatterLayer()
virtual nvinfer1::IScatterLayer::∼IScatterLayer ( ) [protected], [virtual], [default], [noexcept]
9.124.3
Member Function Documentation
9.124.3.1 getAxis()
int32 t nvinfer1::IScatterLayer::getAxis ( ) const [inline], [noexcept]
Get the axis.
9.124.3.2 getMode()
ScatterMode nvinfer1::IScatterLayer::getMode ( ) const [inline], [noexcept]
Get the scatter mode.
See also
setMode()
9.124.3.3 setAxis()
void nvinfer1::IScatterLayer::setAxis (
int32 t axis ) [inline], [noexcept]
Set the axis used by ScatterMode::kELEMENTS.
The axis defaults to 0.
624
Class Documentation
9.124.3.4 setMode()
void nvinfer1::IScatterLayer::setMode (
ScatterMode mode ) [inline], [noexcept]
Set the scatter mode.
See also
getMode()
9.124.4
Member Data Documentation
9.124.4.1 mImpl
apiv::VScatterLayer∗ nvinfer1::IScatterLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.125
nvinfer1::ISelectLayer Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::ISelectLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::ISelectLayer
Protected Member Functions
• virtual ∼ISelectLayer () noexcept=default
Protected Attributes
• apiv::VSelectLayer ∗ mImpl
Additional Inherited Members
9.125.1
Detailed Description
9.126 nvinfer1::IShapeLayer Class Reference
625
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.125.2
Constructor & Destructor Documentation
9.125.2.1 ∼ISelectLayer()
virtual nvinfer1::ISelectLayer::∼ISelectLayer ( ) [protected], [virtual], [default], [noexcept]
9.125.3
Member Data Documentation
9.125.3.1 mImpl
apiv::VSelectLayer∗ nvinfer1::ISelectLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.126
nvinfer1::IShapeLayer Class Reference
Layer type for getting shape of a tensor.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IShapeLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IShapeLayer
Protected Member Functions
• virtual ∼IShapeLayer () noexcept=default
626
Class Documentation
Protected Attributes
• apiv::VShapeLayer ∗ mImpl
Additional Inherited Members
9.126.1
Detailed Description
Layer type for getting shape of a tensor.
This layer sets the output to a 1D tensor of type Int32 with the dimensions of the input tensor.
For example, if the input is a four-dimensional tensor (of any type) with dimensions [2,3,5,7], the output tensor is a
one-dimensional Int32 tensor of length 4 containing the sequence 2, 3, 5, 7.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.126.2
Constructor & Destructor Documentation
9.126.2.1 ∼IShapeLayer()
virtual nvinfer1::IShapeLayer::∼IShapeLayer ( ) [protected], [virtual], [default], [noexcept]
9.126.3
Member Data Documentation
9.126.3.1 mImpl
apiv::VShapeLayer∗ nvinfer1::IShapeLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.127 nvinfer1::IShuffleLayer Class Reference
627
9.127
nvinfer1::IShuffleLayer Class Reference
Layer type for shuffling data.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IShuffleLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IShuffleLayer
Public Member Functions
• void setFirstTranspose (Permutation permutation) noexcept
Set the permutation applied by the first transpose operation.
• Permutation getFirstTranspose () const noexcept
Get the permutation applied by the first transpose operation.
• void setReshapeDimensions (Dims dimensions) noexcept
Set the reshaped dimensions.
• Dims getReshapeDimensions () const noexcept
Get the reshaped dimensions.
• void setSecondTranspose (Permutation permutation) noexcept
Set the permutation applied by the second transpose operation.
• Permutation getSecondTranspose () const noexcept
Get the permutation applied by the second transpose operation.
• void setZeroIsPlaceholder (bool zeroIsPlaceholder) noexcept
Set meaning of 0 in reshape dimensions.
• bool getZeroIsPlaceholder () const noexcept
Get meaning of 0 in reshape dimensions.
• void setInput (int32 t index, ITensor &tensor) noexcept
Append or replace an input of this layer with a specific tensor.
Protected Member Functions
• virtual ∼IShuffleLayer () noexcept=default
Protected Attributes
• apiv::VShuffleLayer ∗ mImpl
628
Class Documentation
9.127.1
Detailed Description
Layer type for shuffling data.
This layer shuffles data by applying in sequence: a transpose operation, a reshape operation and a second transpose
operation. The dimension types of the output are those of the reshape dimension.
The layer has an optional second input. If present, it must be a 1D Int32 shape tensor, and the reshape dimensions are
taken from it.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.127.2
Constructor & Destructor Documentation
9.127.2.1 ∼IShuffleLayer()
virtual nvinfer1::IShuffleLayer::∼IShuffleLayer ( ) [protected], [virtual], [default], [noexcept]
9.127.3
Member Function Documentation
9.127.3.1 getFirstTranspose()
Permutation nvinfer1::IShuffleLayer::getFirstTranspose ( ) const [inline], [noexcept]
Get the permutation applied by the first transpose operation.
Returns
The dimension permutation applied before the reshape.
See also
setFirstTranspose
9.127 nvinfer1::IShuffleLayer Class Reference
629
9.127.3.2 getReshapeDimensions()
Dims nvinfer1::IShuffleLayer::getReshapeDimensions ( ) const [inline], [noexcept]
Get the reshaped dimensions.
Returns
The reshaped dimensions.
If a second input is present and non-null, or setReshapeDimensions has not yet been called, this function returns Dims
with nbDims == -1.
9.127.3.3 getSecondTranspose()
Permutation nvinfer1::IShuffleLayer::getSecondTranspose ( ) const [inline], [noexcept]
Get the permutation applied by the second transpose operation.
Returns
The dimension permutation applied after the reshape.
See also
setSecondTranspose
9.127.3.4 getZeroIsPlaceholder()
bool nvinfer1::IShuffleLayer::getZeroIsPlaceholder ( ) const [inline], [noexcept]
Get meaning of 0 in reshape dimensions.
Returns
true if 0 is placeholder for corresponding input dimension, false if 0 denotes a zero-length dimension.
See also
setZeroIsPlaceholder
9.127.3.5 setFirstTranspose()
void nvinfer1::IShuffleLayer::setFirstTranspose (
Permutation permutation ) [inline], [noexcept]
Set the permutation applied by the first transpose operation.
630
Class Documentation
Parameters
permutation
The dimension permutation applied before the reshape.
The default is the identity permutation.
See also
getFirstTranspose
9.127.3.6 setInput()
void nvinfer1::ILayer::setInput (
int32 t index,
ITensor & tensor ) [inline], [noexcept]
Append or replace an input of this layer with a specific tensor.
Parameters
index
the index of the input to modify.
tensor
the new input tensor Sets the input tensor for the given index. The index must be 0 for a static shuffle
layer. A static shuffle layer is converted to a dynamic shuffle layer by calling setInput with an index 1. A
dynamic shuffle layer cannot be converted back to a static shuffle layer.
For a dynamic shuffle layer, the values 0 and 1 are valid. The indices in the dynamic case are as follows:
• 0: Data or Shape tensor to be shuffled.
• 1: The dimensions for the reshape operation, as a 1D Int32 shape tensor.
If this function is called with the value 1, then the function getNbInputs() changes from returning 1 to 2.
The reshape dimensions are treated identically to how they are treated if set statically via setReshapeDimensions. In
particular, a -1 is treated as a wildcard even if dynamically supplied at runtime, and a 0 is treated as a placeholder
if getZeroIsPlaceholder() = true, which is the default. If the placeholder interpretation of 0 is unwanted because the
runtime dimension should be 0 when the reshape dimension is 0, be sure to call setZeroIsPlacholder(false) on the
IShuffleLayer.
See also
setReshapeDimensions.
9.127 nvinfer1::IShuffleLayer Class Reference
631
9.127.3.7 setReshapeDimensions()
void nvinfer1::IShuffleLayer::setReshapeDimensions (
Dims dimensions ) [inline], [noexcept]
Set the reshaped dimensions.
Parameters
dimensions
The reshaped dimensions.
Two special values can be used as dimensions.
Value 0 copies the corresponding dimension from input. This special value can be used more than once in the dimen-
sions. If number of reshape dimensions is less than input, 0s are resolved by aligning the most significant dimensions
of input.
Value -1 infers that particular dimension by looking at input and rest of the reshape dimensions. Note that only a
maximum of one dimension is permitted to be specified as -1.
The product of the new dimensions must be equal to the product of the old.
If a second input had been used to create this layer, that input is reset to null by this method.
9.127.3.8 setSecondTranspose()
void nvinfer1::IShuffleLayer::setSecondTranspose (
Permutation permutation ) [inline], [noexcept]
Set the permutation applied by the second transpose operation.
Parameters
permutation
The dimension permutation applied after the reshape.
The default is the identity permutation.
The permutation is applied as outputDimensionIndex = permutation.order[inputDimensionIndex], so to permute from
CHW order to HWC order, the required permutation is [1, 2, 0].
See also
getSecondTranspose
|
||
|
|
|