|
|
332
Class Documentation
9.60.3.11 getTensorStrides()
virtual Dims nvinfer1::safe::IExecutionContext::getTensorStrides (
AsciiChar const ∗ tensorName ) const [pure virtual], [noexcept]
Return the strides of the buffer for the given tensor name.
The strides are in units of elements, not components or bytes. For example, for TensorFormat::kHWC8, a stride of one
spans 8 scalars.
Parameters
tensorName
The name of an input or output tensor.
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
Returns
The strides of the buffer for the given tensor name. Dims{-1, {}} will be returned if (1) name is not the name of
an input or output tensor, or (2) name is nullptr, or (3) name exceeds the string length limit.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes
9.60.3.12 operator=() [1/2]
IExecutionContext & nvinfer1::safe::IExecutionContext::operator= (
IExecutionContext &&
) & [delete]
9.60.3.13 operator=() [2/2]
IExecutionContext & nvinfer1::safe::IExecutionContext::operator= (
IExecutionContext const &
) & [delete]
9.60 nvinfer1::safe::IExecutionContext Class Reference
333
9.60.3.14 setDeviceMemory()
virtual void nvinfer1::safe::IExecutionContext::setDeviceMemory (
void ∗const memory ) [pure virtual], [noexcept]
Set the device memory for use by this execution context.
If using enqueueV2() to run the network, The memory is in use from the invocation of enqueueV2() until network
execution is complete. Releasing or otherwise using the memory for other purposes during this time will result in
undefined behavior.
Warning
Do not release or use for other purposes the memory set here during network execution.
See also
safe::ICudaEngine::getDeviceMemorySize() safe::ICudaEngine::createExecutionContextWithoutDeviceMemory()
Usage considerations
• Allowed context for the API call
- Thread-safe: No
9.60.3.15 setErrorBuffer()
virtual void nvinfer1::safe::IExecutionContext::setErrorBuffer (
RuntimeErrorInformation ∗const buffer ) [pure virtual], [noexcept]
Set error buffer output for floating point errors.
The error buffer output must be allocated in device memory and will be used for subsequent calls to enqueueV2.
Checking the contents of the error buffer after inference is the responsibility of the application. The pointer passed
here must have alignment adequate for the RuntimeErrorInformation struct.
Warning
Do not release or use the contents of the error buffer for any other purpose before synchronizing on the
CUDA stream passed to enqueueV2.
Parameters
buffer
The device memory to use as floating point error buffer
334
Class Documentation
See also
getErrorBuffer()
Usage considerations
• Allowed context for the API call
- Thread-safe: No
9.60.3.16 setErrorRecorder()
virtual void nvinfer1::safe::IExecutionContext::setErrorRecorder (
IErrorRecorder ∗const recorder ) [pure virtual], [noexcept]
Set the ErrorRecorder for this interface.
Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function
will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder
with the interface, resulting in a call to decRefCount if a recorder has been registered.
Parameters
recorder
The error recorder to register with this interface.
See also
getErrorRecorder()
Usage considerations
• Allowed context for the API call
- Thread-safe: No
9.60.3.17 setInputConsumedEvent()
virtual bool nvinfer1::safe::IExecutionContext::setInputConsumedEvent (
cudaEvent t event ) [pure virtual], [noexcept]
Mark input as consumed.
Passing event==nullptr removes whatever event was set, if any.
9.60 nvinfer1::safe::IExecutionContext Class Reference
335
Parameters
event
The cuda event that is triggered after all input tensors have been consumed.
Returns
True on success, false if error occurred.
Usage considerations
• Allowed context for the API call
- Thread-safe: No
9.60.3.18 setInputTensorAddress()
virtual bool nvinfer1::safe::IExecutionContext::setInputTensorAddress (
AsciiChar const ∗ tensorName,
void const ∗ data ) [pure virtual], [noexcept]
Set memory address for given input tensor.
An address defaults to nullptr.
Before calling enqueueV3(), each input must have a non-null address.
Parameters
tensorName
The name of an input tensor.
data
The pointer (void const∗) to the const data owned by the user.
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
The pointer must have at least 256-byte alignment.
Returns
True on success, false if (1) name is not the name of an input tensor, or (2) name is nullptr, or (3) name exceeds
the string length limit, or (4) pointer to the const data is nullptr or not aligned.
Usage considerations
336
Class Documentation
• Allowed context for the API call
- Thread-safe: No
9.60.3.19 setName()
virtual void nvinfer1::safe::IExecutionContext::setName (
AsciiChar const ∗const name ) [pure virtual], [noexcept]
Set the name of the execution context.
This method copies the name string.
Warning
Strings passed to the runtime must be 1024 characters or less including NULL terminator and must be
NULL terminated.
See also
getName()
Usage considerations
• Allowed context for the API call
- Thread-safe: No
9.60.3.20 setOutputTensorAddress()
virtual bool nvinfer1::safe::IExecutionContext::setOutputTensorAddress (
AsciiChar const ∗ tensorName,
void ∗ data ) [pure virtual], [noexcept]
Set memory address for given output tensor.
An address defaults to nullptr.
Before calling enqueueV3(), each output must have a non-null address.
Parameters
tensorName
The name of an output tensor.
data
The pointer (void∗) to the data owned by the user.
9.61 nvinfer1::IExprBuilder Class Reference
337
Warning
The string tensorName must be 1024 characters or less including NULL terminator and must be NULL
terminated.
The pointer must have at least 256-byte alignment.
Returns
True on success. Return false if (1) name is not the name of an output tensor, or (2) name is nullptr, or (3) name
exceeds the string length limit, or (4) pointer to data is nullptr or not aligned.
Usage considerations
• Allowed context for the API call
- Thread-safe: No
The documentation for this class was generated from the following file:
• NvInferSafeRuntime.h
9.61
nvinfer1::IExprBuilder Class Reference
#include <NvInferRuntime.h>
Inheritance diagram for nvinfer1::IExprBuilder:
nvinfer1::INoCopy
nvinfer1::IExprBuilder
Public Member Functions
• IDimensionExpr const ∗ constant (int32 t value) noexcept
Return pointer to IDimensionExp for given value.
• IDimensionExpr const ∗ operation (DimensionOperation op, IDimensionExpr const &first, IDimensionExpr
const &second) noexcept
Protected Member Functions
• virtual ∼IExprBuilder () noexcept=default
338
Class Documentation
Protected Attributes
• apiv::VExprBuilder ∗ mImpl
9.61.1
Detailed Description
Object for constructing IDimensionExpr.
There is no public way to construct an IExprBuilder. It appears as an argument to method IPluginV2DynamicExt::getOutputDimensions()
Overrides of that method can use that IExprBuilder argument to construct expressions that define output dimensions
in terms of input dimensions.
Clients should assume that any values constructed by the IExprBuilder are destroyed after IPluginV2DynamicExt::getOutputDimensions(
returns.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
See also
IDimensionExpr
9.61.2
Constructor & Destructor Documentation
9.61.2.1
∼IExprBuilder()
virtual nvinfer1::IExprBuilder::∼IExprBuilder ( ) [protected], [virtual], [default], [noexcept]
9.61.3
Member Function Documentation
9.61.3.1
constant()
IDimensionExpr const ∗ nvinfer1::IExprBuilder::constant (
int32 t value ) [inline], [noexcept]
Return pointer to IDimensionExp for given value.
9.62 nvinfer1::IFillLayer Class Reference
339
9.61.3.2
operation()
IDimensionExpr const ∗ nvinfer1::IExprBuilder::operation (
DimensionOperation op,
IDimensionExpr const & first,
IDimensionExpr const & second ) [inline], [noexcept]
Return pointer to IDimensionExp that represents the given operation applied to first and second. Returns nullptr if op
is not a valid DimensionOperation.
9.61.4
Member Data Documentation
9.61.4.1
mImpl
apiv::VExprBuilder∗ nvinfer1::IExprBuilder::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInferRuntime.h
9.62
nvinfer1::IFillLayer Class Reference
Generate an output tensor with specified mode.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IFillLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IFillLayer
340
Class Documentation
Public Member Functions
• void setDimensions (Dims dimensions) noexcept
Set the output tensor's dimensions.
• Dims getDimensions () const noexcept
Get the output tensor's dimensions.
• void setOperation (FillOperation op) noexcept
Set the fill operation for the layer.
• FillOperation getOperation () const noexcept
Get the fill operation for the layer.
• void setAlpha (double alpha) noexcept
Set the alpha parameter.
• double getAlpha () const noexcept
Get the value of alpha parameter.
• void setBeta (double beta) noexcept
Set the beta parameter.
• double getBeta () const noexcept
Get the value of beta parameter.
• void setInput (int32 t index, ITensor &tensor) noexcept
replace an input of this layer with a specific tensor.
Protected Member Functions
• virtual ∼IFillLayer () noexcept=default
Protected Attributes
• apiv::VFillLayer ∗ mImpl
9.62.1
Detailed Description
Generate an output tensor with specified mode.
The fill layer has two variants, static and dynamic. Static fill specifies its parameters at layer creation time via Dims
and the get/set accessor functions of the IFillLayer. Dynamic fill specifies one or more of its parameters as ITensors, by
using ILayer::setInput to add a corresponding input. The corresponding static parameter is used if an input is missing
or null.
The shape of the output is specified by the parameter Dimension, or if non-null and present, the first input, which
must be a 1D Int32 shape tensor. Thus an application can determine if the IFillLayer has a dynamic output shape based
on whether it has a non-null first input.
Alpha and Beta are treated differently based on the Fill Operation specified. See details in IFillLayer::setAlpha(),
IFillLayer::setBeta(), and IFillLayer::setInput().
A fill layer can produce a shape tensor if the following restrictions are met:
• The FillOperation is kLINSPACE.
• The output is an Int32 or Float tensor within the volume limit of a shape tensor.
• There is at most one input, and if so, that input is input 0.
• If input 0 exists, the length of the output tensor must be computable by constant folding.
9.62 nvinfer1::IFillLayer Class Reference
341
See also
FillOperation
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.62.2
Constructor & Destructor Documentation
9.62.2.1
∼IFillLayer()
virtual nvinfer1::IFillLayer::∼IFillLayer ( ) [protected], [virtual], [default], [noexcept]
9.62.3
Member Function Documentation
9.62.3.1
getAlpha()
double nvinfer1::IFillLayer::getAlpha ( ) const [inline], [noexcept]
Get the value of alpha parameter.
Returns
A double value of alpha.
If the second input is present and non-null, this function returns -1.0.
See also
setAlpha
342
Class Documentation
9.62.3.2
getBeta()
double nvinfer1::IFillLayer::getBeta ( ) const [inline], [noexcept]
Get the value of beta parameter.
Returns
A double value of beta.
If the third input is present and non-null, this function returns -1.0.
See also
setBeta
9.62.3.3
getDimensions()
Dims nvinfer1::IFillLayer::getDimensions ( ) const [inline], [noexcept]
Get the output tensor's dimensions.
Returns
The output tensor's dimensions, or an invalid Dims structure.
If the first input is present and non-null, this function returns a Dims with nbDims = -1.
See also
setDimensions
9.62.3.4
getOperation()
FillOperation nvinfer1::IFillLayer::getOperation ( ) const [inline], [noexcept]
Get the fill operation for the layer.
See also
setOperation(), FillOperation
9.62.3.5
setAlpha()
void nvinfer1::IFillLayer::setAlpha (
double alpha ) [inline], [noexcept]
Set the alpha parameter.
9.62 nvinfer1::IFillLayer Class Reference
343
Parameters
alpha
has different meanings for each
operator:
Operation | Usage kLINSPACE | the start value, defaults to 0.0; kRANDOM UNIFORM | the minimum value, defaults
to 0.0; kRANDOM NORMAL | the mean of the normal distribution, default is 0.0;
If a second input had been used to create this layer, that input is reset to null by this method.
See also
getAlpha
9.62.3.6
setBeta()
void nvinfer1::IFillLayer::setBeta (
double beta ) [inline], [noexcept]
Set the beta parameter.
Parameters
beta
has different meanings for each
operator:
Operation | Usage kLINSPACE | the delta value, defaults to 1.0; kRANDOM UNIFORM | the maximal value, defaults
to 1.0; kRANDOM NORMAL | the standard deviation of the normal distribution, default is 1.0;
If a third input had been used to create this layer, that input is reset to null by this method.
See also
getBeta
9.62.3.7
setDimensions()
void nvinfer1::IFillLayer::setDimensions (
Dims dimensions ) [inline], [noexcept]
Set the output tensor's dimensions.
344
Class Documentation
Parameters
dimensions
The output tensor's dimensions.
If the first input had been used to create this layer, that input is reset to null by this method.
See also
getDimensions
9.62.3.8
setInput()
void nvinfer1::ILayer::setInput (
int32 t index,
ITensor & tensor ) [inline], [noexcept]
replace an input of this layer with a specific tensor.
Parameters
index
the index of the input to set.
tensor
the new input tensor
Indices for kLINSPACE are described as:
• 0: Shape tensor, represents the output tensor's dimensions.
• 1: Start, a scalar, represents the start value.
• 2: Delta, a 1D tensor, length equals to shape tensor's nbDims, represents the delta value for each dimension.
Indices for kRANDOM UNIFORM are described as:
• 0: Shape tensor, represents the output tensor's dimensions.
• 1: Minimum, a scalar, represents the minimum random value.
• 2: Maximum, a scalar, represents the maximal random value.
Indices for kRANDOM NORMAL are described as:
• 0: Shape tensor, represents the output tensor's dimensions.
•
1: Mean, a scalar, represents the mean of the normal distribution,.
9.63 nvinfer1::IFullyConnectedLayer Class Reference
345
• 2: Scale, a scalar, represents the standard deviation of the normal distribution.
Using the corresponding setter resets the input to null.
If either inputs 1 or 2, is non-null, then both must be non-null and have the same data type.
If this function is called for an index greater or equal to getNbInputs(), then afterwards getNbInputs() returns index +
1, and any missing intervening inputs are set to null.
9.62.3.9
setOperation()
void nvinfer1::IFillLayer::setOperation (
FillOperation op ) [inline], [noexcept]
Set the fill operation for the layer.
See also
getOperation(), FillOperation
9.62.4
Member Data Documentation
9.62.4.1
mImpl
apiv::VFillLayer∗ nvinfer1::IFillLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.63
nvinfer1::IFullyConnectedLayer Class Reference
A fully connected layer in a network definition. This layer expects an input tensor of three or more non-batch dimen-
sions. The input is automatically reshaped into an MxV tensor X, where V is a product of the last three dimensions and
M is a product of the remaining dimensions (where the product over 0 dimensions is defined as 1). For example:
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IFullyConnectedLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IFullyConnectedLayer
346
Class Documentation
Public Member Functions
• void setNbOutputChannels (int32 t nbOutputs) noexcept
Set the number of output channels K from the fully connected layer.
• int32 t getNbOutputChannels () const noexcept
Get the number of output channels K from the fully connected layer.
• void setKernelWeights (Weights weights) noexcept
Set the kernel weights, given as a KxC matrix in row-major order.
• Weights getKernelWeights () const noexcept
Get the kernel weights.
• void setBiasWeights (Weights weights) noexcept
Set the bias weights.
• Weights getBiasWeights () const noexcept
Get the bias weights.
• void setInput (int32 t index, ITensor &tensor) noexcept
Append or replace an input of this layer with a specific tensor.
Protected Member Functions
• virtual ∼IFullyConnectedLayer () noexcept=default
Protected Attributes
• apiv::VFullyConnectedLayer ∗ mImpl
9.63.1
Detailed Description
A fully connected layer in a network definition. This layer expects an input tensor of three or more non-batch dimen-
sions. The input is automatically reshaped into an MxV tensor X, where V is a product of the last three dimensions and
M is a product of the remaining dimensions (where the product over 0 dimensions is defined as 1). For example:
• If the input tensor has shape {C, H, W}, then the tensor is reshaped into {1, C∗H∗W}.
• If the input tensor has shape {P, C, H, W}, then the tensor is reshaped into {P, C∗H∗W}.
The layer then performs the following operation:
Y := matmul(X, WˆT) + bias
Where X is the MxV tensor defined above, W is the KxV weight tensor of the layer, and bias is a row vector size K that
is broadcasted to MxK. K is the number of output channels, and configurable via setNbOutputChannels(). If bias is
not specified, it is implicitly 0.
The MxK result Y is then reshaped such that the last three dimensions are {K, 1, 1} and the remaining dimensions
match the dimensions of the input tensor. For example:
• If the input tensor has shape {C, H, W}, then the output tensor will have shape {K, 1, 1}.
• If the input tensor has shape {P, C, H, W}, then the output tensor will have shape {P, K,
1, 1}.
9.63 nvinfer1::IFullyConnectedLayer Class Reference
347
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
Deprecated Deprecated in TensorRT 8.4. Superseded by IMatrixMultiplyLayer.
9.63.2
Constructor & Destructor Documentation
9.63.2.1
∼IFullyConnectedLayer()
virtual nvinfer1::IFullyConnectedLayer::∼IFullyConnectedLayer ( ) [protected], [virtual], [default],
[noexcept]
9.63.3
Member Function Documentation
9.63.3.1
getBiasWeights()
Weights nvinfer1::IFullyConnectedLayer::getBiasWeights ( ) const [inline], [noexcept]
Get the bias weights.
See also
setBiasWeightsWeights()
9.63.3.2
getKernelWeights()
Weights nvinfer1::IFullyConnectedLayer::getKernelWeights ( ) const [inline], [noexcept]
Get the kernel weights.
See also
setKernelWeights()
348
Class Documentation
9.63.3.3
getNbOutputChannels()
int32 t nvinfer1::IFullyConnectedLayer::getNbOutputChannels ( ) const [inline], [noexcept]
Get the number of output channels K from the fully connected layer.
See also
setNbOutputChannels()
9.63.3.4
setBiasWeights()
void nvinfer1::IFullyConnectedLayer::setBiasWeights (
Weights weights ) [inline], [noexcept]
Set the bias weights.
Bias is optional. To omit bias, set the count value in the weights structure to zero.
See also
getBiasWeightsWeights()
9.63.3.5
setInput()
void nvinfer1::ILayer::setInput (
int32 t index,
ITensor & tensor ) [inline], [noexcept]
Append or replace an input of this layer with a specific tensor.
Parameters
index
the index of the input to modify.
tensor
the new input tensor
Only index 0 (data input) is valid, unless explicit-quantization mode is enabled. In explicit-quantization mode, in-
put with index 1 is the kernel-weights tensor, if present. The kernel-weights tensor must be a build-time constant
(computable at build-time via constant-folding) and an output of a dequantize layer. If input index 1 is used then the
kernel-weights parameter must be set to empty Weights.
9.63 nvinfer1::IFullyConnectedLayer Class Reference
349
See also
getKernelWeights(), setKernelWeights()
The indices are as follows:
• 0: The input activation tensor.
• 1: The kernel weights tensor (a constant tensor).
If this function is called with the value 1, then the function getNbInputs() changes from returning 1 to 2.
9.63.3.6
setKernelWeights()
void nvinfer1::IFullyConnectedLayer::setKernelWeights (
Weights weights ) [inline], [noexcept]
Set the kernel weights, given as a KxC matrix in row-major order.
See also
getKernelWeights()
9.63.3.7
setNbOutputChannels()
void nvinfer1::IFullyConnectedLayer::setNbOutputChannels (
int32 t nbOutputs ) [inline], [noexcept]
Set the number of output channels K from the fully connected layer.
If executing this layer on DLA, number of output channels must in the range [1,8192].
See also
getNbOutputChannels()
9.63.4
Member Data Documentation
350
Class Documentation
9.63.4.1
mImpl
apiv::VFullyConnectedLayer∗ nvinfer1::IFullyConnectedLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.64
nvinfer1::IGatherLayer Class Reference
A Gather layer in a network definition. Supports several kinds of gathering.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IGatherLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IGatherLayer
Public Member Functions
• void setGatherAxis (int32 t axis) noexcept
Set the axis used by GatherMode::kELEMENTS and GatherMode::kDEFAULT The axis must be less than the number of
dimensions in the data input. The axis defaults to 0.
• int32 t getGatherAxis () const noexcept
Get the axis to gather on.
• void setNbElementWiseDims (int32 t elementWiseDims) noexcept
Set the number of leading dimensions of indices tensor to be handled elementwise. The gathering of indexing starts from
the dimension of data[NbElementWiseDims:]. The NbElementWiseDims must be less than the Rank of the data input.
• int32 t getNbElementWiseDims () const noexcept
Get the number of leading dimensions of indices tensor to be handled elementwise.
• void setMode (GatherMode mode) noexcept
Set the gather mode.
• GatherMode getMode () const noexcept
Get the gather mode.
Protected Member Functions
• virtual ∼IGatherLayer () noexcept=default
9.64 nvinfer1::IGatherLayer Class Reference
351
Protected Attributes
• apiv::VGatherLayer ∗ mImpl
9.64.1
Detailed Description
A Gather layer in a network definition. Supports several kinds of gathering.
The Gather layer has two input tensors, Data and Indices, and an output tensor Output. Additionally, there are three
parameters: mode, nbElementwiseDims, and axis that control how the indices are interpreted.
• Data is a tensor of rank r >= 1 that stores the values to be gathered in Output.
• Indices is a tensor of rank q that determines which locations in Data to gather.
- GatherMode::kDEFAULT: q >= 0
- GatherMode::kND: q >= 1 and the last dimension of Indices must be a build time constant.
- GatherMode::kELEMENT: q = r
• Output stores the gathered results. Its rank s depends on the mode:
- GatherMode::kDEFAULT: s = q + r - 1 - nbElementwiseDims
- GatherMode::kND: s = q + r - indices.d[q-1] - 1 - nbElementwiseDims
- GatherMode::kELEMENT: s = q = r. The output can be a shape tensor only if the mode is
GatherMode::kDEFAULT.
The dimensions of the output likewise depends on the mode:
GatherMode::kDEFAULT:
First nbElementwiseDims of output are computed by applying broadcast rules to
first nbElementwiseDims of indices and data. Note that nbElementwiseDims <= 1.
Rest of dimensions are computed by copying dimensions of Data, and replacing
the dimension for axis gatherAxis with the dimensions of indices.
GatherMode::kND:
If indices.d[q-1] = r - nbElementwiseDims
output.d = [indices.d[0], ... , indices.d[q-2]]
Else if indices.d[q-1] < r - nbElementWiseDims
output.d = [indices.d[0], ... , indices.d[q-1], data.d[nbElementwiseDims + indices.d[q-1] + q],
data.d[r-1]]
Else
This is build time error
GatherMode::kELEMENT:
The output dimensions match the dimensions of the indices tensor.
The types of Data and Output must be the same, and Indices shall be DataType::kINT32.
How the elements of Data are gathered depends on the mode:
352
Class Documentation
GatherMode::kDEFAULT:
Each index in indices is used to index Data along axis gatherAxis.
GatherMode::kND:
Indices is a rank q integer tensor, best thought of as a rank (q-1) tensor of
indices into data, where each element defines a slice of data
The operation can be formulated as output[i_1, ..., i_{q-1}] = data[indices[i_1, ..., i_{q-1}]]
GatherMode::kELEMENT:
Here "axis" denotes the result of getGatherAxis().
For each element X of indices:
Let J denote a sequence for the subscripts of X
Let K = sequence J with element [axis] replaced by X
output[J] = data[K]
The handling of nbElementWiseDims depends on the mode:
• GatherMode::kDEFAULT: nbElementWiseDims <= 1. Broadcast is supported across the elementwise dimen-
sion if present.
• GatherMode::kND: 0 <= nbElementWiseDims < rank(Data)-1. Broadcast is not supported across the element-
wise dimensions.
• GatherMode::kELEMENT: nbElementWiseDims = 0
Notes:
• For modes GatherMode::kND and GatherMode::kELEMENT, the first nbElementWiseDims dimensions of data
and index must be equal. If not, an error will be reported at build time or run time.
• Only mode GatherMode::kDEFAULT supports an implicit batch dimensions or broadcast on the elementwise
dimensions.
• If an axis of Data has dynamic length, using a negative index for it has undefined behavior.
• No DLA support
• Zero will be stored for OOB access
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.64.2
Constructor & Destructor Documentation
9.64.2.1
∼IGatherLayer()
virtual nvinfer1::IGatherLayer::∼IGatherLayer ( ) [protected], [virtual], [default], [noexcept]
9.64 nvinfer1::IGatherLayer Class Reference
353
9.64.3
Member Function Documentation
9.64.3.1
getGatherAxis()
int32 t nvinfer1::IGatherLayer::getGatherAxis ( ) const [inline], [noexcept]
Get the axis to gather on.
Warning
Undefined behavior when used with GatherMode::kND.
See also
setGatherAxis()
9.64.3.2
getMode()
GatherMode nvinfer1::IGatherLayer::getMode ( ) const [inline], [noexcept]
Get the gather mode.
See also
setMode()
9.64.3.3
getNbElementWiseDims()
int32 t nvinfer1::IGatherLayer::getNbElementWiseDims ( ) const [inline], [noexcept]
Get the number of leading dimensions of indices tensor to be handled elementwise.
See also
setNbElementWiseDims()
9.64.3.4
setGatherAxis()
void nvinfer1::IGatherLayer::setGatherAxis (
int32 t axis ) [inline], [noexcept]
Set the axis used by GatherMode::kELEMENTS and GatherMode::kDEFAULT The axis must be less than the number
of dimensions in the data input. The axis defaults to 0.
354
Class Documentation
Warning
Undefined behavior when used with GatherMode::kND.
See also
getGatherAxis()
9.64.3.5
setMode()
void nvinfer1::IGatherLayer::setMode (
GatherMode mode ) [inline], [noexcept]
Set the gather mode.
See also
getMode()
9.64.3.6
setNbElementWiseDims()
void nvinfer1::IGatherLayer::setNbElementWiseDims (
int32 t elementWiseDims ) [inline], [noexcept]
Set the number of leading dimensions of indices tensor to be handled elementwise. The gathering of indexing starts
from the dimension of data[NbElementWiseDims:]. The NbElementWiseDims must be less than the Rank of the data
input.
Parameters
elementWiseDims
number of dims to be handled as elementwise.
Default: 0
The value of nbElementWiseDims and GatherMode are checked during network validation:
GatherMode::kDEFAULT: nbElementWiseDims must be 0 if there is an implicit batch dimension. It can be 0 or 1 if
there is not an implicit batch dimension. GatherMode::kND: nbElementWiseDims can be between 0 and one less than
rank(data). GatherMode::kELEMENT: nbElementWiseDims must be 0
See also
getNbElementWiseDims()
9.65 nvinfer1::IGpuAllocator Class Reference
355
9.64.4
Member Data Documentation
9.64.4.1
mImpl
apiv::VGatherLayer∗ nvinfer1::IGatherLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.65
nvinfer1::IGpuAllocator Class Reference
Application-implemented class for controlling allocation on the GPU.
#include <NvInferRuntimeBase.h>
Public Member Functions
• virtual void ∗ allocate (uint64 t const size, uint64 t const alignment, AllocatorFlags const flags) noexcept=0
• virtual TRT DEPRECATED void free (void ∗const memory) noexcept=0
• virtual ∼IGpuAllocator ()=default
• IGpuAllocator ()=default
• virtual void ∗ reallocate (void ∗, uint64 t, uint64 t) noexcept
• virtual bool deallocate (void ∗const memory) noexcept
9.65.1
Detailed Description
Application-implemented class for controlling allocation on the GPU.
9.65.2
Constructor & Destructor Documentation
9.65.2.1
∼IGpuAllocator()
virtual nvinfer1::IGpuAllocator::∼IGpuAllocator ( ) [virtual], [default]
Destructor declared virtual as general good practice for a class with virtual methods. TensorRT never calls the destruc-
tor for an IGpuAllocator defined by the application.
356
Class Documentation
9.65.2.2
IGpuAllocator()
nvinfer1::IGpuAllocator::IGpuAllocator ( ) [default]
9.65.3
Member Function Documentation
9.65.3.1
allocate()
virtual void ∗ nvinfer1::IGpuAllocator::allocate (
uint64 t const size,
uint64 t const alignment,
AllocatorFlags const flags ) [pure virtual], [noexcept]
A thread-safe callback implemented by the application to handle acquisition of GPU memory.
Parameters
size
The size of the memory required.
alignment
The required alignment of memory. Alignment will be zero or a power of 2 not exceeding the
alignment guaranteed by cudaMalloc. Thus this allocator can be safely implemented with
cudaMalloc/cudaFree. An alignment value of zero indicates any alignment is acceptable.
flags
Reserved for future use. In the current release, 0 will be passed.
If an allocation request of size 0 is made, nullptr should be returned.
If an allocation request cannot be satisfied, nullptr should be returned.
Note
The implementation must guarantee thread safety for concurrent allocate/free/reallocate/deallocate requests.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads.
9.65.3.2
deallocate()
virtual bool nvinfer1::IGpuAllocator::deallocate (
void ∗const memory ) [inline], [virtual], [noexcept]
A thread-safe callback implemented by the application to handle release of GPU memory.
TensorRT may pass a nullptr to this function if it was previously returned by allocate().
9.65 nvinfer1::IGpuAllocator Class Reference
357
Parameters
memory
The acquired memory.
Returns
True if the acquired memory is released successfully.
Note
The implementation must guarantee thread safety for concurrent allocate/free/reallocate/deallocate requests.
If user-implemented free() might hit an error condition, the user should override deallocate() as the primary
implementation and override free() to call deallocate() for backwards compatibility.
See also
free()
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads.
9.65.3.3
free()
virtual TRT DEPRECATED void nvinfer1::IGpuAllocator::free (
void ∗const memory ) [pure virtual], [noexcept]
A thread-safe callback implemented by the application to handle release of GPU memory.
TensorRT may pass a nullptr to this function if it was previously returned by allocate().
Parameters
memory
The acquired memory.
Note
The implementation must guarantee thread safety for concurrent allocate/free/reallocate/deallocate requests.
358
Class Documentation
See also
deallocate()
Deprecated Deprecated in TensorRT 8.0. Superseded by deallocate.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads.
9.65.3.4
reallocate()
virtual void ∗ nvinfer1::IGpuAllocator::reallocate (
void ∗ ,
uint64 t ,
uint64 t
) [inline], [virtual], [noexcept]
A thread-safe callback implemented by the application to resize an existing allocation.
Only allocations which were allocated with AllocatorFlag::kRESIZABLE will be resized.
Options are one of:
• resize in place leaving min(oldSize, newSize) bytes unchanged and return the original address
• move min(oldSize, newSize) bytes to a new location of sufficient size and return its address
• return nullptr, to indicate that the request could not be fulfilled.
If nullptr is returned, TensorRT will assume that resize() is not implemented, and that the allocation at baseAddr is still
valid.
This method is made available for use cases where delegating the resize strategy to the application provides an oppor-
tunity to improve memory management. One possible implementation is to allocate a large virtual device buffer and
progressively commit physical memory with cuMemMap. CU MEM ALLOC GRANULARITY RECOMMENDED
is suggested in this case.
TensorRT may call realloc to increase the buffer by relatively small amounts.
Parameters
baseAddr
the address of the original allocation.
alignment
The alignment used by the original allocation.
newSize
The new memory size required.
9.66 nvinfer1::IGridSampleLayer Class Reference
359
Returns
the address of the reallocated memory
Note
The implementation must guarantee thread safety for concurrent allocate/free/reallocate/deallocate requests.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads.
The documentation for this class was generated from the following file:
• NvInferRuntimeBase.h
9.66
nvinfer1::IGridSampleLayer Class Reference
A GridSample layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IGridSampleLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IGridSampleLayer
Public Member Functions
• void setInterpolationMode (InterpolationMode mode) noexcept
Set the grid sample interpolation mode.
• InterpolationMode getInterpolationMode () const noexcept
Get the grid sample interpolation mode.
• void setAlignCorners (bool alignCorners) noexcept
Set the align corners mode.
• bool getAlignCorners () const noexcept
Get the align corners mode.
• bool setSampleMode (SampleMode mode) noexcept
Set the sample mode.
• SampleMode getSampleMode () const noexcept
Get the sample mode.
360
Class Documentation
Protected Member Functions
• virtual ∼IGridSampleLayer () noexcept=default
Protected Attributes
• apiv::VGridSampleLayer ∗ mImpl
9.66.1
Detailed Description
A GridSample layer in a network definition.
This layer uses an input tensor and a grid tensor to produce an interpolated output tensor. The input and grid
tensors must be shape tensors of rank 4. The only supported SampleMode values are SampleMode::kCLAMP,
SampleMode::kFILL, and SampleMode::kREFLECT.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.66.2
Constructor & Destructor Documentation
9.66.2.1
∼IGridSampleLayer()
virtual nvinfer1::IGridSampleLayer::∼IGridSampleLayer ( ) [protected], [virtual], [default],
[noexcept]
9.66.3
Member Function Documentation
9.66.3.1
getAlignCorners()
bool nvinfer1::IGridSampleLayer::getAlignCorners ( ) const [inline], [noexcept]
Get the align corners mode.
See also
setAlignCorners()
Returns
The value specified by setAlignCorners(), or false otherwise.
9.66 nvinfer1::IGridSampleLayer Class Reference
361
9.66.3.2
getInterpolationMode()
InterpolationMode nvinfer1::IGridSampleLayer::getInterpolationMode ( ) const [inline], [noexcept]
Get the grid sample interpolation mode.
See also
setInterpolationMode()
Returns
The value specified by setInterpolationMode, or InterpolationMode::kLINEAR otherwise.
9.66.3.3
getSampleMode()
SampleMode nvinfer1::IGridSampleLayer::getSampleMode ( ) const [inline], [noexcept]
Get the sample mode.
See also
setSampleMode()
Returns
the value specified by a successful call to setSampleMode(), or SampleMode::kFILL otherwise.
9.66.3.4
setAlignCorners()
void nvinfer1::IGridSampleLayer::setAlignCorners (
bool alignCorners ) [inline], [noexcept]
Set the align corners mode.
See also
getAlignCorners()
362
Class Documentation
9.66.3.5
setInterpolationMode()
void nvinfer1::IGridSampleLayer::setInterpolationMode (
InterpolationMode mode ) [inline], [noexcept]
Set the grid sample interpolation mode.
See also
getInterpolationMode()
9.66.3.6
setSampleMode()
bool nvinfer1::IGridSampleLayer::setSampleMode (
SampleMode mode ) [inline], [noexcept]
Set the sample mode.
See also
getSampleMode()
Returns
true if layer's sample mode was set to mode, false otherwise.
9.66.4
Member Data Documentation
9.66.4.1
mImpl
apiv::VGridSampleLayer∗ nvinfer1::IGridSampleLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.67 nvinfer1::IHostMemory Class Reference
363
9.67
nvinfer1::IHostMemory Class Reference
Class to handle library allocated memory that is accessible to the user.
#include <NvInferRuntime.h>
Inheritance diagram for nvinfer1::IHostMemory:
nvinfer1::INoCopy
nvinfer1::IHostMemory
Public Member Functions
• virtual ∼IHostMemory () noexcept=default
• void ∗ data () const noexcept
A pointer to the raw data that is owned by the library.
• std::size t size () const noexcept
The size in bytes of the data that was allocated.
• DataType type () const noexcept
The type of the memory that was allocated.
• TRT DEPRECATED void destroy () noexcept
Protected Attributes
• apiv::VHostMemory ∗ mImpl
Additional Inherited Members
9.67.1
Detailed Description
Class to handle library allocated memory that is accessible to the user.
The memory allocated via the host memory object is owned by the library and will be de-allocated when the destroy
method is called.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.67.2
Constructor & Destructor Documentation
364
Class Documentation
9.67.2.1
∼IHostMemory()
virtual nvinfer1::IHostMemory::∼IHostMemory ( ) [virtual], [default], [noexcept]
9.67.3
Member Function Documentation
9.67.3.1
data()
void ∗ nvinfer1::IHostMemory::data ( ) const [inline], [noexcept]
A pointer to the raw data that is owned by the library.
9.67.3.2
destroy()
TRT DEPRECATED void nvinfer1::IHostMemory::destroy ( ) [inline], [noexcept]
Destroy the allocated memory.
Deprecated Deprecated in TRT 8.0. Superseded by delete.
Warning
Calling destroy on a managed pointer will result in a double-free error.
9.67.3.3
size()
std::size t nvinfer1::IHostMemory::size ( ) const [inline], [noexcept]
The size in bytes of the data that was allocated.
9.67.3.4
type()
DataType nvinfer1::IHostMemory::type ( ) const [inline], [noexcept]
The type of the memory that was allocated.
9.68 nvinfer1::IIdentityLayer Class Reference
365
9.67.4
Member Data Documentation
9.67.4.1
mImpl
apiv::VHostMemory∗ nvinfer1::IHostMemory::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInferRuntime.h
9.68
nvinfer1::IIdentityLayer Class Reference
A layer that represents the identity function.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IIdentityLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IIdentityLayer
Protected Member Functions
• virtual ∼IIdentityLayer () noexcept=default
Protected Attributes
• apiv::VIdentityLayer ∗ mImpl
366
Class Documentation
Additional Inherited Members
9.68.1
Detailed Description
A layer that represents the identity function.
If the output type is explicitly specified via setOutputType, IIdentityLayer can be used to convert from one type to
another. Other than conversions between the same type (kFLOAT -> kFLOAT for example), the only valid conversions
are:
(kFLOAT | kHALF | kINT32 | kBOOL) -> (kFLOAT | kHALF | kINT32 | kBOOL)
(kFLOAT | kHALF) -> kUINT8
kUINT8 -> (kFLOAT | kHALF)
Conversion also happens implicitly, without calling setOutputType, if the output tensor is a network output.
Two types are compatible if they are identical, or are both in {kFLOAT, kHALF}. Implicit conversion between incom-
patible types, i.e. without using setOutputType, is recognized as incorrect as of TensorRT 8.4, but is retained for API
compatibility within TensorRT 8.x releases. In a future major release the behavior will change to record an error if
the network output tensor type is incompatible with the layer output type. E.g., implicit conversion from kFLOAT to
kINT32 will not be allowed, and instead such a conversion will require calling setOutputType(DataType::kINT32).
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.68.2
Constructor & Destructor Documentation
9.68.2.1
∼IIdentityLayer()
virtual nvinfer1::IIdentityLayer::∼IIdentityLayer ( ) [protected], [virtual], [default], [noexcept]
9.68.3
Member Data Documentation
9.68.3.1
mImpl
apiv::VIdentityLayer∗ nvinfer1::IIdentityLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.69 nvinfer1::IIfConditional Class Reference
367
9.69
nvinfer1::IIfConditional Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IIfConditional:
nvinfer1::INoCopy
nvinfer1::IIfConditional
Public Member Functions
• IConditionLayer ∗ setCondition (ITensor &condition) noexcept
Set the condition tensor for this If-Conditional construct.
• IIfConditionalOutputLayer ∗ addOutput (ITensor &trueSubgraphOutput, ITensor &falseSubgraphOutput) noex-
cept
Add an If-conditional output.
• IIfConditionalInputLayer ∗ addInput (ITensor &input) noexcept
Add an If-conditional input.
• void setName (char const ∗name) noexcept
Set the name of the conditional.
• char const ∗ getName () const noexcept
Return the name of the conditional.
Protected Member Functions
• virtual ∼IIfConditional () noexcept=default
Protected Attributes
• apiv::VIfConditional ∗ mImpl
9.69.1
Detailed Description
Helper for constructing conditionally-executed subgraphs.
An If-conditional conditionally executes part of the network according to the following pseudo-code:
If condition is true then: output = trueSubgraph(trueInputs); Else output = falseSubgraph(falseInputs); Emit output
Condition is a 0D boolean tensor (representing a scalar). trueSubgraph represents a network subgraph that is executed
when condition is evaluated to True. falseSubgraph represents a network subgraph that is executed when condition is
evaluated to False.
The following constraints apply to If-conditionals:
• Both the trueSubgraph and falseSubgraph must be defined.
• The number of output tensors in both subgraphs is the same.
• The type and shape of each output tensor from true/false subgraphs are the same.
368
Class Documentation
9.69.2
Constructor & Destructor Documentation
9.69.2.1
∼IIfConditional()
virtual nvinfer1::IIfConditional::∼IIfConditional ( ) [protected], [virtual], [default], [noexcept]
9.69.3
Member Function Documentation
9.69.3.1
addInput()
IIfConditionalInputLayer ∗ nvinfer1::IIfConditional::addInput (
ITensor & input ) [inline], [noexcept]
Add an If-conditional input.
Parameters
input
An input to the conditional that can be used by either or both of the conditional’s subgraphs.
See also
IIfConditionalInputLayer
9.69.3.2
addOutput()
IIfConditionalOutputLayer ∗ nvinfer1::IIfConditional::addOutput (
ITensor & trueSubgraphOutput,
ITensor & falseSubgraphOutput ) [inline], [noexcept]
Add an If-conditional output.
Parameters
trueSubgraphOutput
The output of the subgraph executed when the conditional evaluates to true.
falseSubgraphOutput
The output of the subgraph executed when the conditional evaluates to false.
9.69 nvinfer1::IIfConditional Class Reference
369
Each output layer of an IIfConditional represents a single output of either the true-subgraph or the false-subgraph of
an IIfConditional, depending on which subgraph was executed.
See also
IIfConditionalOutputLayer
9.69.3.3
getName()
char const ∗ nvinfer1::IIfConditional::getName ( ) const [inline], [noexcept]
Return the name of the conditional.
See also
setName()
9.69.3.4
setCondition()
IConditionLayer ∗ nvinfer1::IIfConditional::setCondition (
ITensor & condition ) [inline], [noexcept]
Set the condition tensor for this If-Conditional construct.
Parameters
condition
The condition tensor that will determine which subgraph to execute.
condition tensor must be a 0D execution tensor (scalar) with type DataType::kBOOL.
See also
IConditionLayer
9.69.3.5
setName()
void nvinfer1::IIfConditional::setName (
char const ∗ name ) [inline], [noexcept]
Set the name of the conditional.
The name is used in error diagnostics. This method copies the name string.
370
Class Documentation
Warning
The string name must be null-terminated, and be at most 4096 bytes including the terminator.
See also
getName()
9.69.4
Member Data Documentation
9.69.4.1
mImpl
apiv::VIfConditional∗ nvinfer1::IIfConditional::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.70
nvinfer1::IIfConditionalBoundaryLayer Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IIfConditionalBoundaryLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IIfConditionalBoundaryLayer
nvinfer1::IConditionLayer
nvinfer1::IIfConditionalInputLayer
nvinfer1::IIfConditionalOutputLayer
Public Member Functions
• IIfConditional ∗ getConditional () const noexcept
Return pointer to the IIfConditional associated with this boundary layer.
Protected Member Functions
• virtual ∼IIfConditionalBoundaryLayer () noexcept=default
9.70 nvinfer1::IIfConditionalBoundaryLayer Class Reference
371
Protected Attributes
• apiv::VConditionalBoundaryLayer ∗ mBoundary
9.70.1
Detailed Description
This is a base class for Conditional boundary layers.
Boundary layers are used to demarcate the boundaries of Conditionals.
9.70.2
Constructor & Destructor Documentation
9.70.2.1
∼IIfConditionalBoundaryLayer()
virtual nvinfer1::IIfConditionalBoundaryLayer::∼IIfConditionalBoundaryLayer ( ) [protected],
[virtual], [default], [noexcept]
9.70.3
Member Function Documentation
9.70.3.1
getConditional()
IIfConditional ∗ nvinfer1::IIfConditionalBoundaryLayer::getConditional ( ) const [inline], [noexcept]
Return pointer to the IIfConditional associated with this boundary layer.
9.70.4
Member Data Documentation
9.70.4.1
mBoundary
apiv::VConditionalBoundaryLayer∗ nvinfer1::IIfConditionalBoundaryLayer::mBoundary [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
372
Class Documentation
9.71
nvinfer1::IIfConditionalInputLayer Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IIfConditionalInputLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IIfConditionalBoundaryLayer
nvinfer1::IIfConditionalInputLayer
Protected Member Functions
• virtual ∼IIfConditionalInputLayer () noexcept=default
Protected Attributes
• apiv::VConditionalInputLayer ∗ mImpl
Additional Inherited Members
9.71.1
Detailed Description
This layer represents an input to an IIfConditional.
9.71.2
Constructor & Destructor Documentation
9.71.2.1
∼IIfConditionalInputLayer()
virtual nvinfer1::IIfConditionalInputLayer::∼IIfConditionalInputLayer ( ) [protected], [virtual],
[default], [noexcept]
9.71.3
Member Data Documentation
9.72 nvinfer1::IIfConditionalOutputLayer Class Reference
373
9.71.3.1
mImpl
apiv::VConditionalInputLayer∗ nvinfer1::IIfConditionalInputLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.72
nvinfer1::IIfConditionalOutputLayer Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IIfConditionalOutputLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IIfConditionalBoundaryLayer
nvinfer1::IIfConditionalOutputLayer
Protected Member Functions
• virtual ∼IIfConditionalOutputLayer () noexcept=default
Protected Attributes
• apiv::VConditionalOutputLayer ∗ mImpl
Additional Inherited Members
9.72.1
Detailed Description
This layer represents an output of an IIfConditional.
An IIfConditionalOutputLayer has exactly one output.
9.72.2
Constructor & Destructor Documentation
374
Class Documentation
9.72.2.1
∼IIfConditionalOutputLayer()
virtual nvinfer1::IIfConditionalOutputLayer::∼IIfConditionalOutputLayer ( ) [protected], [virtual],
[default], [noexcept]
9.72.3
Member Data Documentation
9.72.3.1
mImpl
apiv::VConditionalOutputLayer∗ nvinfer1::IIfConditionalOutputLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.73
nvinfer1::IInt8Calibrator Class Reference
Application-implemented interface for calibration.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IInt8Calibrator:
nvinfer1::IInt8Calibrator
nvinfer1::IInt8EntropyCalibrator
nvinfer1::IInt8EntropyCalibrator2
nvinfer1::IInt8LegacyCalibrator
nvinfer1::IInt8MinMaxCalibrator
Public Member Functions
• virtual int32 t getBatchSize () const noexcept=0
Get the batch size used for calibration batches.
• virtual bool getBatch (void ∗bindings[ ], char const ∗names[ ], int32 t nbBindings) noexcept=0
Get a batch of input for calibration.
• virtual void const ∗ readCalibrationCache (std::size t &length) noexcept=0
Load a calibration cache.
• virtual void writeCalibrationCache (void const ∗ptr, std::size t length) noexcept=0
Save a calibration cache.
• virtual CalibrationAlgoType getAlgorithm () noexcept=0
Get the algorithm used by this calibrator.
• virtual ∼IInt8Calibrator () noexcept=default
9.73 nvinfer1::IInt8Calibrator Class Reference
375
9.73.1
Detailed Description
Application-implemented interface for calibration.
Calibration is a step performed by the builder when deciding suitable scale factors for 8-bit inference.
It must also provide a method for retrieving representative images which the calibration process can use to examine
the distribution of activations. It may optionally implement a method for caching the calibration result for reuse on
subsequent runs.
9.73.2
Constructor & Destructor Documentation
9.73.2.1
∼IInt8Calibrator()
virtual nvinfer1::IInt8Calibrator::∼IInt8Calibrator ( ) [virtual], [default], [noexcept]
9.73.3
Member Function Documentation
9.73.3.1
getAlgorithm()
virtual CalibrationAlgoType nvinfer1::IInt8Calibrator::getAlgorithm ( ) [pure virtual], [noexcept]
Get the algorithm used by this calibrator.
Returns
The algorithm used by the calibrator.
Implemented in nvinfer1::IInt8EntropyCalibrator, nvinfer1::IInt8EntropyCalibrator2, nvinfer1::IInt8MinMaxCalibrator,
and nvinfer1::IInt8LegacyCalibrator.
9.73.3.2
getBatch()
virtual bool nvinfer1::IInt8Calibrator::getBatch (
void ∗ bindings[ ],
char const ∗ names[ ],
int32 t nbBindings ) [pure virtual], [noexcept]
Get a batch of input for calibration.
The batch size of the input must match the batch size returned by getBatchSize().
376
Class Documentation
Parameters
bindings
An array of pointers to device memory that must be updated to point to device memory containing
each network input data.
names
The names of the network input for each pointer in the binding array.
nbBindings
The number of pointers in the bindings array.
Returns
False if there are no more batches for calibration.
See also
getBatchSize()
9.73.3.3
getBatchSize()
virtual int32 t nvinfer1::IInt8Calibrator::getBatchSize ( ) const [pure virtual], [noexcept]
Get the batch size used for calibration batches.
Returns
The batch size.
9.73.3.4
readCalibrationCache()
virtual void const ∗ nvinfer1::IInt8Calibrator::readCalibrationCache (
std::size t & length ) [pure virtual], [noexcept]
Load a calibration cache.
Calibration is potentially expensive, so it can be useful to generate the calibration data once, then use it on subsequent
builds of the network. The cache includes the regression cutoff and quantile values used to generate it, and will not be
used if these do not batch the settings of the current calibrator. However, the network should also be recalibrated if its
structure changes, or the input data set changes, and it is the responsibility of the application to ensure this.
Parameters
length
The length of the cached data, that should be set by the called function. If there is no data, this should be
zero.
9.74 nvinfer1::IInt8EntropyCalibrator Class Reference
377
Returns
A pointer to the cache, or nullptr if there is no data.
9.73.3.5
writeCalibrationCache()
virtual void nvinfer1::IInt8Calibrator::writeCalibrationCache (
void const ∗ ptr,
std::size t length ) [pure virtual], [noexcept]
Save a calibration cache.
Parameters
ptr
A pointer to the data to cache.
length
The length in bytes of the data to cache.
See also
readCalibrationCache()
The documentation for this class was generated from the following file:
• NvInfer.h
9.74
nvinfer1::IInt8EntropyCalibrator Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IInt8EntropyCalibrator:
nvinfer1::IInt8Calibrator
nvinfer1::IInt8EntropyCalibrator
Public Member Functions
• CalibrationAlgoType getAlgorithm () noexcept override
• virtual ∼IInt8EntropyCalibrator () noexcept=default
378
Class Documentation
9.74.1
Detailed Description
Entropy calibrator. This is the Legacy Entropy calibrator. It is less complicated than the legacy calibrator and produces
better results.
9.74.2
Constructor & Destructor Documentation
9.74.2.1
∼IInt8EntropyCalibrator()
virtual nvinfer1::IInt8EntropyCalibrator::∼IInt8EntropyCalibrator ( ) [virtual], [default], [noexcept]
9.74.3
Member Function Documentation
9.74.3.1
getAlgorithm()
CalibrationAlgoType nvinfer1::IInt8EntropyCalibrator::getAlgorithm ( ) [inline], [override],
[virtual], [noexcept]
Signal that this is the entropy calibrator.
Implements nvinfer1::IInt8Calibrator.
The documentation for this class was generated from the following file:
• NvInfer.h
9.75
nvinfer1::IInt8EntropyCalibrator2 Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IInt8EntropyCalibrator2:
nvinfer1::IInt8Calibrator
nvinfer1::IInt8EntropyCalibrator2
9.76 nvinfer1::IInt8LegacyCalibrator Class Reference
379
Public Member Functions
• CalibrationAlgoType getAlgorithm () noexcept override
• virtual ∼IInt8EntropyCalibrator2 () noexcept=default
9.75.1
Detailed Description
Entropy calibrator 2. This is the preferred calibrator. This is the required calibrator for DLA, as it supports per
activation tensor scaling.
9.75.2
Constructor & Destructor Documentation
9.75.2.1
∼IInt8EntropyCalibrator2()
virtual nvinfer1::IInt8EntropyCalibrator2::∼IInt8EntropyCalibrator2 ( ) [virtual], [default],
[noexcept]
9.75.3
Member Function Documentation
9.75.3.1
getAlgorithm()
CalibrationAlgoType nvinfer1::IInt8EntropyCalibrator2::getAlgorithm ( ) [inline], [override],
[virtual], [noexcept]
Signal that this is the entropy calibrator 2.
Implements nvinfer1::IInt8Calibrator.
The documentation for this class was generated from the following file:
• NvInfer.h
9.76
nvinfer1::IInt8LegacyCalibrator Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IInt8LegacyCalibrator:
nvinfer1::IInt8Calibrator
nvinfer1::IInt8LegacyCalibrator
380
Class Documentation
Public Member Functions
• CalibrationAlgoType getAlgorithm () noexcept override
• virtual double getQuantile () const noexcept=0
The quantile (between 0 and 1) that will be used to select the region maximum when the quantile method is in use.
• virtual double getRegressionCutoff () const noexcept=0
The fraction (between 0 and 1) of the maximum used to define the regression cutoff when using regression to determine
the region maximum.
• virtual void const ∗ readHistogramCache (std::size t &length) noexcept=0
Load a histogram.
• virtual void writeHistogramCache (void const ∗ptr, std::size t length) noexcept=0
Save a histogram cache.
• virtual ∼IInt8LegacyCalibrator () noexcept=default
9.76.1
Detailed Description
Legacy calibrator left for backward compatibility with TensorRT 2.0. This calibrator requires user parameterization,
and is provided as a fallback option if the other calibrators yield poor results.
9.76.2
Constructor & Destructor Documentation
9.76.2.1
∼IInt8LegacyCalibrator()
virtual nvinfer1::IInt8LegacyCalibrator::∼IInt8LegacyCalibrator ( ) [virtual], [default], [noexcept]
9.76.3
Member Function Documentation
9.76.3.1
getAlgorithm()
CalibrationAlgoType nvinfer1::IInt8LegacyCalibrator::getAlgorithm ( ) [inline], [override], [virtual],
[noexcept]
Signal that this is the legacy calibrator.
Implements nvinfer1::IInt8Calibrator.
9.76 nvinfer1::IInt8LegacyCalibrator Class Reference
381
9.76.3.2
getQuantile()
virtual double nvinfer1::IInt8LegacyCalibrator::getQuantile ( ) const [pure virtual], [noexcept]
The quantile (between 0 and 1) that will be used to select the region maximum when the quantile method is in use.
See the user guide for more details on how the quantile is used.
9.76.3.3
getRegressionCutoff()
virtual double nvinfer1::IInt8LegacyCalibrator::getRegressionCutoff ( ) const [pure virtual],
[noexcept]
The fraction (between 0 and 1) of the maximum used to define the regression cutoff when using regression to determine
the region maximum.
See the user guide for more details on how the regression cutoff is used
9.76.3.4
readHistogramCache()
virtual void const ∗ nvinfer1::IInt8LegacyCalibrator::readHistogramCache (
std::size t & length ) [pure virtual], [noexcept]
Load a histogram.
Histogram generation is potentially expensive, so it can be useful to generate the histograms once, then use them when
exploring the space of calibrations. The histograms should be regenerated if the network structure changes, or the input
data set changes, and it is the responsibility of the application to ensure this.
Parameters
length
The length of the cached data, that should be set by the called function. If there is no data, this should be
zero.
Returns
A pointer to the cache, or nullptr if there is no data.
9.76.3.5
writeHistogramCache()
virtual void nvinfer1::IInt8LegacyCalibrator::writeHistogramCache (
void const ∗ ptr,
std::size t length ) [pure virtual], [noexcept]
Save a histogram cache.
382
Class Documentation
Parameters
ptr
A pointer to the data to cache.
length
The length in bytes of the data to cache.
See also
readHistogramCache()
The documentation for this class was generated from the following file:
• NvInfer.h
9.77
nvinfer1::IInt8MinMaxCalibrator Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IInt8MinMaxCalibrator:
nvinfer1::IInt8Calibrator
nvinfer1::IInt8MinMaxCalibrator
Public Member Functions
• CalibrationAlgoType getAlgorithm () noexcept override
• virtual ∼IInt8MinMaxCalibrator () noexcept=default
9.77.1
Detailed Description
MinMax Calibrator. It supports per activation tensor scaling.
9.77.2
Constructor & Destructor Documentation
9.77.2.1
∼IInt8MinMaxCalibrator()
virtual nvinfer1::IInt8MinMaxCalibrator::∼IInt8MinMaxCalibrator ( ) [virtual], [default], [noexcept]
9.78 nvinfer1::IIteratorLayer Class Reference
383
9.77.3
Member Function Documentation
9.77.3.1
getAlgorithm()
CalibrationAlgoType nvinfer1::IInt8MinMaxCalibrator::getAlgorithm ( ) [inline], [override], [virtual],
[noexcept]
Signal that this is the MinMax Calibrator.
Implements nvinfer1::IInt8Calibrator.
The documentation for this class was generated from the following file:
• NvInfer.h
9.78
nvinfer1::IIteratorLayer Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::IIteratorLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::ILoopBoundaryLayer
nvinfer1::IIteratorLayer
Public Member Functions
• void setAxis (int32 t axis) noexcept
Set axis to iterate over.
• int32 t getAxis () const noexcept
Get axis being iterated over.
• void setReverse (bool reverse) noexcept
• bool getReverse () const noexcept
True if and only if reversing input.
384
Class Documentation
Protected Member Functions
• virtual ∼IIteratorLayer () noexcept=default
Protected Attributes
• apiv::VIteratorLayer ∗ mImpl
9.78.1
Constructor & Destructor Documentation
9.78.1.1
∼IIteratorLayer()
virtual nvinfer1::IIteratorLayer::∼IIteratorLayer ( ) [protected], [virtual], [default], [noexcept]
9.78.2
Member Function Documentation
9.78.2.1
getAxis()
int32 t nvinfer1::IIteratorLayer::getAxis ( ) const [inline], [noexcept]
Get axis being iterated over.
9.78.2.2
getReverse()
bool nvinfer1::IIteratorLayer::getReverse ( ) const [inline], [noexcept]
True if and only if reversing input.
9.78.2.3
setAxis()
void nvinfer1::IIteratorLayer::setAxis (
int32 t axis ) [inline], [noexcept]
Set axis to iterate over.
9.79 nvinfer1::ILayer Class Reference
385
9.78.2.4
setReverse()
void nvinfer1::IIteratorLayer::setReverse (
bool reverse ) [inline], [noexcept]
For reverse=false, the layer is equivalent to addGather(tensor, I, 0) where I is a scalar tensor containing the loop iteration
number. For reverse=true, the layer is equivalent to addGather(tensor, M-1-I, 0) where M is the trip count computed
from TripLimits of kind kCOUNT. The default is reverse=false.
9.78.3
Member Data Documentation
9.78.3.1
mImpl
apiv::VIteratorLayer∗ nvinfer1::IIteratorLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.79
nvinfer1::ILayer Class Reference
Base class for all layer classes in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::ILayer:
386
Class Documentation
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::IActivationLayer
nvinfer1::IAssertionLayer
nvinfer1::ICastLayer
nvinfer1::IConcatenationLayer
nvinfer1::IConstantLayer
nvinfer1::IConvolutionLayer
nvinfer1::IDeconvolutionLayer
nvinfer1::IDequantizeLayer
nvinfer1::IEinsumLayer
nvinfer1::IElementWiseLayer
nvinfer1::IFillLayer
nvinfer1::IFullyConnectedLayer
nvinfer1::IGatherLayer
nvinfer1::IGridSampleLayer
nvinfer1::IIdentityLayer
nvinfer1::IIfConditionalBoundaryLayer
nvinfer1::ILRNLayer
nvinfer1::ILoopBoundaryLayer
nvinfer1::IMatrixMultiplyLayer
nvinfer1::INMSLayer
nvinfer1::INonZeroLayer
nvinfer1::INormalizationLayer
nvinfer1::IOneHotLayer
nvinfer1::IPaddingLayer
nvinfer1::IParametricReLULayer
nvinfer1::IPluginV2Layer
nvinfer1::IPoolingLayer
nvinfer1::IQuantizeLayer
nvinfer1::IRNNv2Layer
nvinfer1::IRaggedSoftMaxLayer
nvinfer1::IReduceLayer
nvinfer1::IResizeLayer
nvinfer1::IReverseSequenceLayer
nvinfer1::IScaleLayer
nvinfer1::IScatterLayer
nvinfer1::ISelectLayer
nvinfer1::IShapeLayer
nvinfer1::IShuffleLayer
nvinfer1::ISliceLayer
nvinfer1::ISoftMaxLayer
nvinfer1::ITopKLayer
nvinfer1::IUnaryLayer
Public Member Functions
• LayerType getType () const noexcept
Return the type of a layer.
• void setName (char const ∗name) noexcept
Set the name of a layer.
• char const ∗ getName () const noexcept
Return the name of a layer.
• int32 t getNbInputs () const noexcept
Get the number of inputs of a layer.
• ITensor ∗ getInput (int32 t index) const noexcept
Get the layer input corresponding to the given index.
• int32 t getNbOutputs () const noexcept
Get the number of outputs of a layer.
• ITensor ∗ getOutput (int32 t index) const noexcept
Get the layer output corresponding to the given index.
• void setInput (int32 t index, ITensor &tensor) noexcept
Replace an input of this layer with a specific tensor.
• void setPrecision (DataType dataType) noexcept
Set the computational precision of this layer.
9.79 nvinfer1::ILayer Class Reference
387
• DataType getPrecision () const noexcept
get the computational precision of this layer
• bool precisionIsSet () const noexcept
whether the computational precision has been set for this layer
• void resetPrecision () noexcept
reset the computational precision for this layer
• void setOutputType (int32 t index, DataType dataType) noexcept
Set the output type of this layer.
• DataType getOutputType (int32 t index) const noexcept
get the output type of this layer
• bool outputTypeIsSet (int32 t index) const noexcept
whether the output type has been set for this layer
• void resetOutputType (int32 t index) noexcept
reset the output type for this layer
• void setMetadata (char const ∗metadata) noexcept
Set the metadata for this layer.
• char const ∗ getMetadata () const noexcept
Get the metadata of the layer.
Protected Member Functions
• virtual ∼ILayer () noexcept=default
Protected Attributes
• apiv::VLayer ∗ mLayer
9.79.1
Detailed Description
Base class for all layer classes in a network definition.
Warning
Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.
9.79.2
Constructor & Destructor Documentation
9.79.2.1
∼ILayer()
virtual nvinfer1::ILayer::∼ILayer ( ) [protected], [virtual], [default], [noexcept]
388
Class Documentation
9.79.3
Member Function Documentation
9.79.3.1
getInput()
ITensor ∗ nvinfer1::ILayer::getInput (
int32 t index ) const [inline], [noexcept]
Get the layer input corresponding to the given index.
Parameters
index
The index of the input tensor.
Returns
The input tensor, or nullptr if the index is out of range or the tensor is optional (ISliceLayer and IRNNv2Layer).
9.79.3.2
getMetadata()
char const ∗ nvinfer1::ILayer::getMetadata ( ) const [inline], [noexcept]
Get the metadata of the layer.
Returns
The metadata as a null-terminated C-style string. If setMetadata() has not been called, an empty string ”” will be
returned as a default value.
See also
setMetadata()
9.79.3.3
getName()
char const ∗ nvinfer1::ILayer::getName ( ) const [inline], [noexcept]
Return the name of a layer.
See also
setName()
9.79 nvinfer1::ILayer Class Reference
389
9.79.3.4
getNbInputs()
int32 t nvinfer1::ILayer::getNbInputs ( ) const [inline], [noexcept]
Get the number of inputs of a layer.
9.79.3.5
getNbOutputs()
int32 t nvinfer1::ILayer::getNbOutputs ( ) const [inline], [noexcept]
Get the number of outputs of a layer.
9.79.3.6
getOutput()
ITensor ∗ nvinfer1::ILayer::getOutput (
int32 t index ) const [inline], [noexcept]
Get the layer output corresponding to the given index.
Returns
The indexed output tensor, or nullptr if the index is out of range or the tensor is optional (IRNNv2Layer).
9.79.3.7
getOutputType()
DataType nvinfer1::ILayer::getOutputType (
int32 t index ) const [inline], [noexcept]
get the output type of this layer
Parameters
index
the index of the output
Returns
the output precision. If no precision has been set, DataType::kFLOAT will be returned, unless the output type is
inherently DataType::kINT32.
390
Class Documentation
See also
getOutputType() outputTypeIsSet() resetOutputType()
9.79.3.8
getPrecision()
DataType nvinfer1::ILayer::getPrecision ( ) const [inline], [noexcept]
get the computational precision of this layer
Returns
the computational precision
See also
setPrecision() precisionIsSet() resetPrecision()
9.79.3.9
getType()
LayerType nvinfer1::ILayer::getType ( ) const [inline], [noexcept]
Return the type of a layer.
See also
LayerType
9.79.3.10 outputTypeIsSet()
bool nvinfer1::ILayer::outputTypeIsSet (
int32 t index ) const [inline], [noexcept]
whether the output type has been set for this layer
Parameters
index
the index of the output
9.79 nvinfer1::ILayer Class Reference
391
Returns
whether the output type has been explicitly set
See also
setOutputType() getOutputType() resetOutputType()
9.79.3.11 precisionIsSet()
bool nvinfer1::ILayer::precisionIsSet ( ) const [inline], [noexcept]
whether the computational precision has been set for this layer
Returns
whether the computational precision has been explicitly set
See also
setPrecision() getPrecision() resetPrecision()
9.79.3.12 resetOutputType()
void nvinfer1::ILayer::resetOutputType (
int32 t index ) [inline], [noexcept]
reset the output type for this layer
Parameters
index
the index of the output
See also
setOutputType() getOutputType() outputTypeIsSet()
392
Class Documentation
9.79.3.13 resetPrecision()
void nvinfer1::ILayer::resetPrecision ( ) [inline], [noexcept]
reset the computational precision for this layer
See also
setPrecision() getPrecision() precisionIsSet()
9.79.3.14 setInput()
void nvinfer1::ILayer::setInput (
int32 t index,
ITensor & tensor ) [inline], [noexcept]
Replace an input of this layer with a specific tensor.
Parameters
index
the index of the input to modify.
tensor
the new input tensor
Except for IFillLayer, ILoopOutputLayer, INMSLayer, IResizeLayer, IShuffleLayer, and ISliceLayer, this method
cannot change the number of inputs to a layer. The index argument must be less than the value of getNbInputs().
See comments for overloads of setInput() for layers with special behavior.
9.79.3.15 setMetadata()
void nvinfer1::ILayer::setMetadata (
char const ∗ metadata ) [inline], [noexcept]
Set the metadata for this layer.
The metadata is emitted in the JSON returned by IEngineInspector with ProfilingVerbosity set to kDETAILED.
Parameters
metadata
The per-layer metadata.
9.79 nvinfer1::ILayer Class Reference
393
Warning
The string name must be null-terminated and be at most 4096 bytes including the terminator.
See also
getMetadata()
getLayerInformation()
9.79.3.16 setName()
void nvinfer1::ILayer::setName (
char const ∗ name ) [inline], [noexcept]
Set the name of a layer.
This method copies the name string.
Warning
The string name must be null-terminated, and be at most 4096 bytes including the terminator.
See also
getName()
9.79.3.17 setOutputType()
void nvinfer1::ILayer::setOutputType (
int32 t index,
DataType dataType ) [inline], [noexcept]
Set the output type of this layer.
Setting the output type constrains TensorRT to choose implementations which generate output data with the given type.
If it is not set, TensorRT will select output type based on layer computational precision. TensorRT could still choose
non-conforming output type based on fastest implementation. To force choosing the requested output type, set exactly
one of the following flags, which differ in what happens if no such implementation exists:
• BuilderFlag::kOBEY PRECISION CONSTRAINTS - build fails with an error message.
• BuilderFlag::kPREFER PRECISION CONSTRAINTS - TensorRT falls back to an implementation with a non-
conforming output type.
394
Class Documentation
In case layer precision is not specified, or falling back, the output type depends on the chosen implementation, based
on performance considerations and the flags specified to the builder.
This method cannot be used to set the data type of the second output tensor of the TopK layer. The data type of the
second output tensor of the topK layer is always Int32. Also the output type of all layers that are shape operations must
be DataType::kINT32, and all attempts to set the output type to some other data type will be ignored except for issuing
an error message.
Note that the layer output type is generally not identical to the data type of the output tensor, as TensorRT may insert
implicit reformatting operations to convert the former to the latter. Calling layer->setOutputType(i, type) has no effect
on the data type of the i-th output tensor of layer, and users need to call layer->getOutput(i)->setType(type) to change
the tensor data type. This is particularly relevant if the tensor is marked as a network output, since only setType() [but
not setOutputType()] will affect the data representation in the corresponding output binding.
Parameters
index
the index of the output to set
dataType
the type of the output
See also
getOutputType() outputTypeIsSet() resetOutputType()
9.79.3.18 setPrecision()
void nvinfer1::ILayer::setPrecision (
DataType dataType ) [inline], [noexcept]
Set the computational precision of this layer.
Setting the precision allows TensorRT to choose an implementation which run at this computational precision. Tensor←↩
RT could still choose a non-conforming fastest implementation that ignores the requested precision. To force choosing
an implementation with the requested precision, set exactly one of the following flags, which differ in what happens if
no such implementation exists:
• BuilderFlag::kOBEY PRECISION CONSTRAINTS - build fails with an error message.
• BuilderFlag::kPREFER PRECISION CONSTRAINTS - TensorRT falls back to an implementation without the
requested precision.
If precision is not set, or falling back, TensorRT will select the layer computational precision and layer input type based
on global performance considerations and the flags specified to the builder.
For a IIdentityLayer: If it casts to/from float/half/int8/uint8, the precision must be one of those types, otherwise it must
be either the input or output type.
9.80 nvinfer1::ILogger Class Reference
395
Parameters
dataType
the computational precision.
See also
getPrecision() precisionIsSet() resetPrecision()
9.79.4
Member Data Documentation
9.79.4.1
mLayer
apiv::VLayer∗ nvinfer1::ILayer::mLayer [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.80
nvinfer1::ILogger Class Reference
Application-implemented logging interface for the builder, refitter and runtime.
#include <NvInferRuntimeBase.h>
Public Types
• enum class Severity : int32 t {
kINTERNAL ERROR = 0 , kERROR = 1 , kWARNING = 2 , kINFO = 3 ,
kVERBOSE = 4 }
Public Member Functions
• virtual void log (Severity severity, AsciiChar const ∗msg) noexcept=0
• ILogger ()=default
• virtual ∼ILogger ()=default
396
Class Documentation
9.80.1
Detailed Description
Application-implemented logging interface for the builder, refitter and runtime.
The logger used to create an instance of IBuilder, IRuntime or IRefitter is used for all objects created through that
interface. The logger should be valid until all objects created are released.
The Logger object implementation must be thread safe. All locking and synchronization is pushed to the interface
implementation and TensorRT does not hold any synchronization primitives when calling the interface functions.
9.80.2
Member Enumeration Documentation
9.80.2.1
Severity
enum class nvinfer1::ILogger::Severity : int32 t [strong]
The severity corresponding to a log message.
Enumerator
kINTERNAL ERROR
An internal error has occurred. Execution is unrecoverable.
kERROR
An application error has occurred.
kWARNING
An application error has been discovered, but TensorRT has recovered or fallen back to
a default.
kINFO
Informational messages with instructional information.
kVERBOSE
Verbose messages with debugging information.
9.80.3
Constructor & Destructor Documentation
9.80.3.1
ILogger()
nvinfer1::ILogger::ILogger ( ) [default]
9.80.3.2
∼ILogger()
virtual nvinfer1::ILogger::∼ILogger ( ) [virtual], [default]
9.81 nvinfer1::ILoggerFinder Class Reference
397
9.80.4
Member Function Documentation
9.80.4.1
log()
virtual void nvinfer1::ILogger::log (
Severity severity,
AsciiChar const ∗ msg ) [pure virtual], [noexcept]
A callback implemented by the application to handle logging messages;
Parameters
severity
The severity of the message.
msg
A null-terminated log message.
Usage considerations
• Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when
multiple execution contexts are used during runtime, or if the same logger is used for multiple runtimes,
builders, or refitters.
The documentation for this class was generated from the following file:
• NvInferRuntimeBase.h
9.81
nvinfer1::ILoggerFinder Class Reference
A virtual base class to find a logger. Allows a plugin to find an instance of a logger if it needs to emit a log message. A
pointer to an instance of this class is passed to a plugin shared library on initialization when that plugin is serialized as
part of a version-compatible plan. See the plugin chapter in the developer guide for details.
#include <NvInferRuntime.h>
Public Member Functions
• virtual ILogger ∗ findLogger ()=0
Get the logger used by the engine or execution context which called the plugin method.
398
Class Documentation
Protected Member Functions
• virtual ∼ILoggerFinder ()=default
9.81.1
Detailed Description
A virtual base class to find a logger. Allows a plugin to find an instance of a logger if it needs to emit a log message. A
pointer to an instance of this class is passed to a plugin shared library on initialization when that plugin is serialized as
part of a version-compatible plan. See the plugin chapter in the developer guide for details.
9.81.2
Constructor & Destructor Documentation
9.81.2.1
∼ILoggerFinder()
virtual nvinfer1::ILoggerFinder::∼ILoggerFinder ( ) [protected], [virtual], [default]
9.81.3
Member Function Documentation
9.81.3.1
findLogger()
virtual ILogger ∗ nvinfer1::ILoggerFinder::findLogger ( ) [pure virtual]
Get the logger used by the engine or execution context which called the plugin method.
Warning
Must be called from the thread in which the plugin method was called.
Returns
A pointer to the logger.
The documentation for this class was generated from the following file:
• NvInferRuntime.h
9.82 nvinfer1::ILoop Class Reference
399
9.82
nvinfer1::ILoop Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::ILoop:
nvinfer1::INoCopy
nvinfer1::ILoop
Public Member Functions
• IRecurrenceLayer ∗ addRecurrence (ITensor &initialValue) noexcept
Create a recurrence layer for this loop with initialValue as its first input.
• ITripLimitLayer ∗ addTripLimit (ITensor &tensor, TripLimit limit) noexcept
Add a trip-count limiter, based on the given tensor.
• IIteratorLayer ∗ addIterator (ITensor &tensor, int32 t axis=0, bool reverse=false) noexcept
Return layer that subscripts tensor by loop iteration.
• ILoopOutputLayer ∗ addLoopOutput (ITensor &tensor, LoopOutput outputKind, int32 t axis=0) noexcept
Make an output for this loop, based on the given tensor.
• void setName (char const ∗name) noexcept
Set the name of the loop.
• char const ∗ getName () const noexcept
Return the name of the loop.
Protected Member Functions
• virtual ∼ILoop () noexcept=default
Protected Attributes
• apiv::VLoop ∗ mImpl
9.82.1
Detailed Description
Helper for creating a recurrent subgraph.
An ILoop cannot be added to an INetworkDefinition where hasImplicitBatchDimensions() returns true.
9.82.2
Constructor & Destructor Documentation
400
Class Documentation
9.82.2.1
∼ILoop()
virtual nvinfer1::ILoop::∼ILoop ( ) [protected], [virtual], [default], [noexcept]
9.82.3
Member Function Documentation
9.82.3.1
addIterator()
IIteratorLayer ∗ nvinfer1::ILoop::addIterator (
ITensor & tensor,
int32 t axis = 0,
bool reverse = false ) [inline], [noexcept]
Return layer that subscripts tensor by loop iteration.
For reverse=false, this is equivalent to addGather(tensor, I, 0) where I is a scalar tensor containing the loop iteration
number. For reverse=true, this is equivalent to addGather(tensor, M-1-I, 0) where M is the trip count computed from
TripLimits of kind kCOUNT.
9.82.3.2
addLoopOutput()
ILoopOutputLayer ∗ nvinfer1::ILoop::addLoopOutput (
ITensor & tensor,
LoopOutput outputKind,
int32 t axis = 0 ) [inline], [noexcept]
Make an output for this loop, based on the given tensor.
axis is the axis for concatenation (if using outputKind of kCONCATENATE or kREVERSE).
If outputKind is kCONCATENATE or kREVERSE, a second input specifying the concatenation dimension must be
added via method ILoopOutputLayer::setInput.
9.82.3.3
addRecurrence()
IRecurrenceLayer ∗ nvinfer1::ILoop::addRecurrence (
ITensor & initialValue ) [inline], [noexcept]
Create a recurrence layer for this loop with initialValue as its first input.
IRecurrenceLayer requires exactly two inputs. The 2nd input must be added, via method IRecurrenceLayer::set←↩
Input(1,...) before an Engine can be built.
9.82 nvinfer1::ILoop Class Reference
401
9.82.3.4
addTripLimit()
ITripLimitLayer ∗ nvinfer1::ILoop::addTripLimit (
ITensor & tensor,
TripLimit limit ) [inline], [noexcept]
Add a trip-count limiter, based on the given tensor.
There may be at most one kCOUNT and one kWHILE limiter for a loop. When both trip limits exist, the loop exits
when the count is reached or condition is falsified. It is an error to not add at least one trip limiter.
For kCOUNT, the input tensor must be available before the loop starts.
For kWHILE, the input tensor must be the output of a subgraph that contains only layers that are not ITripLimitLayer,
IIteratorLayer or ILoopOutputLayer. Any IRecurrenceLayers in the subgraph must belong to the same loop as the
ITripLimitLayer. A trivial example of this rule is that the input to the kWHILE is the output of an IRecurrenceLayer
for the same loop.
9.82.3.5
getName()
char const ∗ nvinfer1::ILoop::getName ( ) const [inline], [noexcept]
Return the name of the loop.
See also
setName()
9.82.3.6
setName()
void nvinfer1::ILoop::setName (
char const ∗ name ) [inline], [noexcept]
Set the name of the loop.
The name is used in error diagnostics. This method copies the name string.
Warning
The string name must be null-terminated, and be at most 4096 bytes including the terminator.
See also
getName()
402
Class Documentation
9.82.4
Member Data Documentation
9.82.4.1
mImpl
apiv::VLoop∗ nvinfer1::ILoop::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.83
nvinfer1::ILoopBoundaryLayer Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::ILoopBoundaryLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::ILoopBoundaryLayer
nvinfer1::IIteratorLayer
nvinfer1::ILoopOutputLayer
nvinfer1::IRecurrenceLayer
nvinfer1::ITripLimitLayer
Public Member Functions
• ILoop ∗ getLoop () const noexcept
Return pointer to ILoop associated with this boundary layer.
Protected Member Functions
• virtual ∼ILoopBoundaryLayer () noexcept=default
Protected Attributes
• apiv::VLoopBoundaryLayer ∗ mBoundary
9.83.1
Constructor & Destructor Documentation
9.84 nvinfer1::ILoopOutputLayer Class Reference
403
9.83.1.1
∼ILoopBoundaryLayer()
virtual nvinfer1::ILoopBoundaryLayer::∼ILoopBoundaryLayer ( ) [protected], [virtual], [default],
[noexcept]
9.83.2
Member Function Documentation
9.83.2.1
getLoop()
ILoop ∗ nvinfer1::ILoopBoundaryLayer::getLoop ( ) const [inline], [noexcept]
Return pointer to ILoop associated with this boundary layer.
9.83.3
Member Data Documentation
9.83.3.1
mBoundary
apiv::VLoopBoundaryLayer∗ nvinfer1::ILoopBoundaryLayer::mBoundary [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.84
nvinfer1::ILoopOutputLayer Class Reference
#include <NvInfer.h>
Inheritance diagram for nvinfer1::ILoopOutputLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::ILoopBoundaryLayer
nvinfer1::ILoopOutputLayer
404
Class Documentation
Public Member Functions
• LoopOutput getLoopOutput () const noexcept
• void setAxis (int32 t axis) noexcept
Set where to insert the contenation axis. Ignored if getLoopOutput() is kLAST VALUE.
• int32 t getAxis () const noexcept
Get axis being concatenated over.
• void setInput (int32 t index, ITensor &tensor) noexcept
Append or replace an input of this layer with a specific tensor.
Protected Member Functions
• virtual ∼ILoopOutputLayer () noexcept=default
Protected Attributes
• apiv::VLoopOutputLayer ∗ mImpl
9.84.1
Detailed Description
An ILoopOutputLayer is the sole way to get output from a loop.
The first input tensor must be defined inside the loop; the output tensor is outside the loop. The second input tensor, if
present, must be defined outside the loop.
If getLoopOutput() is kLAST VALUE, a single input must be provided, and that input must from a IRecurrenceLayer
in the same loop.
If getLoopOutput() is kCONCATENATE or kREVERSE, a second input must be provided. The second input must be
a 0D shape tensor, defined before the loop commences, that specifies the concatenation length of the output.
The output tensor has j more dimensions than the input tensor, where j == 0 if getLoopOutput() is kLAST VALUE j
== 1 if getLoopOutput() is kCONCATENATE or kREVERSE.
9.84.2
Constructor & Destructor Documentation
9.84.2.1
∼ILoopOutputLayer()
virtual nvinfer1::ILoopOutputLayer::∼ILoopOutputLayer ( ) [protected], [virtual], [default],
[noexcept]
9.84 nvinfer1::ILoopOutputLayer Class Reference
405
9.84.3
Member Function Documentation
9.84.3.1
getAxis()
int32 t nvinfer1::ILoopOutputLayer::getAxis ( ) const [inline], [noexcept]
Get axis being concatenated over.
9.84.3.2
getLoopOutput()
LoopOutput nvinfer1::ILoopOutputLayer::getLoopOutput ( ) const [inline], [noexcept]
9.84.3.3
setAxis()
void nvinfer1::ILoopOutputLayer::setAxis (
int32 t axis ) [inline], [noexcept]
Set where to insert the contenation axis. Ignored if getLoopOutput() is kLAST VALUE.
For example, if the input tensor has dimensions [b,c,d], and getLoopOutput() is kCONCATENATE, the output has
four dimensions. Let a be the value of the second input. setAxis(0) causes the output to have dimensions [a,b,c,d].
setAxis(1) causes the output to have dimensions [b,a,c,d]. setAxis(2) causes the output to have dimensions [b,c,a,d].
setAxis(3) causes the output to have dimensions [b,c,d,a]. Default is axis is 0.
9.84.3.4
setInput()
void nvinfer1::ILayer::setInput (
int32 t index,
ITensor & tensor ) [inline], [noexcept]
Append or replace an input of this layer with a specific tensor.
Parameters
index
the index of the input to modify.
tensor
the new input tensor Sets the input tensor for the given index. The index must be 0 for a kLAST VALUE
loop output layer. Loop output layer is converted to a kCONCATENATE or kREVERSE loop output
layer by calling setInput with an index 1. A kCONCATENATE or kREVERSE loop output layer cannot
be converted back to a kLAST VALUE loop output layer.
406
Class Documentation
For a kCONCATENATE or kREVERSE loop output layer, the values 0 and 1 are valid. The indices in the k←↩
CONCATENATE or kREVERSE cases are as follows:
• 0: Contribution to the output tensor. The contribution must come from inside the loop.
• 1: The concatenation length scalar value, must come from outside the loop, as a 0D Int32 shape tensor.
If this function is called with the value 1, then the function getNbInputs() changes from returning 1 to 2.
9.84.4
Member Data Documentation
9.84.4.1
mImpl
apiv::VLoopOutputLayer∗ nvinfer1::ILoopOutputLayer::mImpl [protected]
The documentation for this class was generated from the following file:
• NvInfer.h
9.85
nvinfer1::ILRNLayer Class Reference
A LRN layer in a network definition.
#include <NvInfer.h>
Inheritance diagram for nvinfer1::ILRNLayer:
nvinfer1::INoCopy
nvinfer1::ILayer
nvinfer1::ILRNLayer
|
||
|
|
|