NVIDIA TensorRT 8.6.11 API Reference for DRIVE OS (May 2023)

9.85 nvinfer1::ILRNLayer Class Reference

407

Public Member Functions

• void setWindowSize (int32 t windowSize) noexcept

Set the LRN window size.

• int32 t getWindowSize () const noexcept

Get the LRN window size.

• void setAlpha (float alpha) noexcept

Set the LRN alpha value.

• float getAlpha () const noexcept

Get the LRN alpha value.

• void setBeta (float beta) noexcept

Set the LRN beta value.

• float getBeta () const noexcept

Get the LRN beta value.

• void setK (float k) noexcept

Set the LRN K value.

• float getK () const noexcept

Get the LRN K value.

Protected Member Functions

• virtual ∼ILRNLayer () noexcept=default

Protected Attributes

• apiv::VLRNLayer ∗ mImpl

9.85.1

Detailed Description

A LRN layer in a network definition.

The output size is the same as the input size.

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.85.2

Constructor & Destructor Documentation

9.85.2.1

∼ILRNLayer()

virtual nvinfer1::ILRNLayer::∼ILRNLayer ( ) [protected], [virtual], [default], [noexcept]

410

Class Documentation

9.85.3.7

setK()

void nvinfer1::ILRNLayer::setK (

float k ) [inline], [noexcept]

Set the LRN K value.

The valid range is [1e-5, 1e10].

See also

getK()

9.85.3.8

setWindowSize()

void nvinfer1::ILRNLayer::setWindowSize (

int32 t windowSize ) [inline], [noexcept]

Set the LRN window size.

The window size must be odd and in the range of [1, 15].

If executing this layer on the DLA, only values in the set, [3, 5, 7, 9], are valid.

See also

setWindowStride()

9.85.4

Member Data Documentation

9.85.4.1

mImpl

apiv::VLRNLayer∗ nvinfer1::ILRNLayer::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.86 nvinfer1::IMatrixMultiplyLayer Class Reference

411

9.86

nvinfer1::IMatrixMultiplyLayer Class Reference

Layer that represents a Matrix Multiplication.

#include <NvInfer.h>

Inheritance diagram for nvinfer1::IMatrixMultiplyLayer:

nvinfer1::INoCopy

nvinfer1::ILayer

nvinfer1::IMatrixMultiplyLayer

Public Member Functions

• void setOperation (int32 t index, MatrixOperation op) noexcept

Set the operation for an input tensor.

• MatrixOperation getOperation (int32 t index) const noexcept

Get the operation for an input tensor.

Protected Member Functions

• virtual ∼IMatrixMultiplyLayer () noexcept=default

Protected Attributes

• apiv::VMatrixMultiplyLayer ∗ mImpl

9.86.1

Detailed Description

Layer that represents a Matrix Multiplication.

Let A be op(getInput(0)) and B be op(getInput(1)) where op(x) denotes the corresponding MatrixOperation.

When A and B are matrices or vectors, computes the inner product A ∗ B:

matrix * matrix -> matrix

matrix * vector -> vector

vector * matrix -> vector

vector * vector -> scalar

Inputs of higher rank are treated as collections of matrices or vectors. The output will be a corresponding collection of

matrices, vectors, or scalars.

For a dimension that is not one of the matrix or vector dimensions: If the dimension is 1 for one of the tensors but not

the other tensor, the former tensor is broadcast along that dimension to match the dimension of the latter tensor. The

number of these extra dimensions for A and B must match.

412

Class Documentation

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.86.2

Constructor & Destructor Documentation

9.86.2.1

∼IMatrixMultiplyLayer()

virtual nvinfer1::IMatrixMultiplyLayer::∼IMatrixMultiplyLayer ( ) [protected], [virtual], [default],

[noexcept]

9.86.3

Member Function Documentation

9.86.3.1

getOperation()

MatrixOperation nvinfer1::IMatrixMultiplyLayer::getOperation (

int32 t index ) const [inline], [noexcept]

Get the operation for an input tensor.

Parameters

index

Input tensor number (0 or 1).

See also

setOperation()

9.86.3.2

setOperation()

void nvinfer1::IMatrixMultiplyLayer::setOperation (

int32 t index,

MatrixOperation op ) [inline], [noexcept]

Set the operation for an input tensor.

9.87 nvinfer1::INetworkDefinition Class Reference

413

Parameters

index

Input tensor number (0 or 1).

New operation.

See also

getOperation()

9.86.4

Member Data Documentation

9.86.4.1

mImpl

apiv::VMatrixMultiplyLayer∗ nvinfer1::IMatrixMultiplyLayer::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.87

nvinfer1::INetworkDefinition Class Reference

A network definition for input to the builder.

#include <NvInfer.h>

Inheritance diagram for nvinfer1::INetworkDefinition:

nvinfer1::INoCopy

nvinfer1::INetworkDefinition

414

Class Documentation

Public Member Functions

•

virtual ∼INetworkDefinition () noexcept=default

•

ITensor ∗ addInput (char const ∗name, DataType type, Dims dimensions) noexcept

Add an input tensor to the network.

•

void markOutput (ITensor &tensor) noexcept

Mark a tensor as a network output.

•

TRT DEPRECATED IConvolutionLayer ∗ addConvolution (ITensor &input, int32 t nbOutputMaps, DimsHW

kernelSize, Weights kernelWeights, Weights biasWeights) noexcept

Add a convolution layer to the network.

•

TRT DEPRECATED IFullyConnectedLayer ∗ addFullyConnected (ITensor &input, int32 t nbOutputs, Weights

kernelWeights, Weights biasWeights) noexcept

Add a fully connected layer to the network.

•

IActivationLayer ∗ addActivation (ITensor &input, ActivationType type) noexcept

Add an activation layer to the network.

•

TRT DEPRECATED IPoolingLayer ∗ addPooling (ITensor &input, PoolingType type, DimsHW windowSize)

noexcept

Add a pooling layer to the network.

•

ILRNLayer ∗ addLRN (ITensor &input, int32 t window, float alpha, float beta, float k) noexcept

Add a LRN layer to the network.

•

IScaleLayer ∗ addScale (ITensor &input, ScaleMode mode, Weights shift, Weights scale, Weights power) noex-

cept

Add a Scale layer to the network.

•

ISoftMaxLayer ∗ addSoftMax (ITensor &input) noexcept

Add a SoftMax layer to the network.

•

IConcatenationLayer ∗ addConcatenation (ITensor ∗const ∗inputs, int32 t nbInputs) noexcept

Add a concatenation layer to the network.

•

TRT DEPRECATED IDeconvolutionLayer ∗ addDeconvolution (ITensor &input, int32 t nbOutputMaps,

DimsHW kernelSize, Weights kernelWeights, Weights biasWeights) noexcept

Add a deconvolution layer to the network.

•

IElementWiseLayer ∗ addElementWise (ITensor &input1, ITensor &input2, ElementWiseOperation op) noex-

cept

Add an elementwise layer to the network.

•

IUnaryLayer ∗ addUnary (ITensor &input, UnaryOperation operation) noexcept

Add a unary layer to the network.

•

TRT DEPRECATED IPaddingLayer ∗ addPadding (ITensor &input, DimsHW prePadding, DimsHW post←↩

Padding) noexcept

Add a padding layer to the network.

•

IShuffleLayer ∗ addShuffle (ITensor &input) noexcept

Add a shuffle layer to the network.

•

IOneHotLayer ∗ addOneHot (ITensor &indices, ITensor &values, ITensor &depth, int32 t axis) noexcept

Add a OneHot layer to the network.

•

int32 t getNbLayers () const noexcept

Get the number of layers in the network.

•

ILayer ∗ getLayer (int32 t index) const noexcept

Get the layer specified by the given index.

•

int32 t getNbInputs () const noexcept

Get the number of inputs in the network.

9.87 nvinfer1::INetworkDefinition Class Reference

415

•

ITensor ∗ getInput (int32 t index) const noexcept

Get the input tensor specified by the given index.

•

int32 t getNbOutputs () const noexcept

Get the number of outputs in the network.

•

ITensor ∗ getOutput (int32 t index) const noexcept

Get the output tensor specified by the given index.

•

TRT DEPRECATED void destroy () noexcept

Destroy this INetworkDefinition object.

•

IReduceLayer ∗ addReduce (ITensor &input, ReduceOperation operation, uint32 t reduceAxes, bool keep←↩

Dimensions) noexcept

Add a reduce layer to the network.

•

ITopKLayer ∗ addTopK (ITensor &input, TopKOperation op, int32 t k, uint32 t reduceAxes) noexcept

Add a TopK layer to the network.

•

IGatherLayer ∗ addGather (ITensor &data, ITensor &indices, int32 t axis) noexcept

Add gather with mode GatherMode::kDEFAULT and specified axis and nbElementWiseDims=0.

•

IGatherLayer ∗ addGatherV2 (ITensor &data, ITensor &indices, GatherMode mode) noexcept

Add gather with specified mode, axis=0 and nbElementWiseDims=0.

•

IRaggedSoftMaxLayer ∗ addRaggedSoftMax (ITensor &input, ITensor &bounds) noexcept

Add a RaggedSoftMax layer to the network.

•

IMatrixMultiplyLayer ∗ addMatrixMultiply (ITensor &input0, MatrixOperation op0, ITensor &input1,

MatrixOperation op1) noexcept

Add a MatrixMultiply layer to the network.

•

INonZeroLayer ∗ addNonZero (ITensor &input) noexcept

Add a nonzero layer to the network.

•

IConstantLayer ∗ addConstant (Dims dimensions, Weights weights) noexcept

Add a constant layer to the network.

•

TRT DEPRECATED IRNNv2Layer ∗ addRNNv2 (ITensor &input, int32 t layerCount, int32 t hiddenSize,

int32 t maxSeqLen, RNNOperation op) noexcept

Add an layerCount deep RNN layer to the network with hiddenSize internal states that can take a batch with fixed

or variable sequence lengths.

•

IIdentityLayer ∗ addIdentity (ITensor &input) noexcept

Add an identity layer.

•

ICastLayer ∗ addCast (ITensor &input, DataType toType) noexcept

Add a cast layer.

•

void removeTensor (ITensor &tensor) noexcept

remove a tensor from the network definition.

•

void unmarkOutput (ITensor &tensor) noexcept

unmark a tensor as a network output.

•

IPluginV2Layer ∗ addPluginV2 (ITensor ∗const ∗inputs, int32 t nbInputs, IPluginV2 &plugin) noexcept

Add a plugin layer to the network using the IPluginV2 interface.

•

ISliceLayer ∗ addSlice (ITensor &input, Dims start, Dims size, Dims stride) noexcept

Add a slice layer to the network.

•

void setName (char const ∗name) noexcept

Sets the name of the network.

•

char const ∗ getName () const noexcept

Returns the name associated with the network.

•

IShapeLayer ∗ addShape (ITensor &input) noexcept

Add a shape layer to the network.

416

Class Documentation

•

bool hasImplicitBatchDimension () const noexcept

Query whether the network was created with an implicit batch dimension.

•

bool markOutputForShapes (ITensor &tensor) noexcept

Enable tensor's value to be computed by IExecutionContext::getShapeBinding.

•

bool unmarkOutputForShapes (ITensor &tensor) noexcept

Undo markOutputForShapes.

•

IParametricReLULayer ∗ addParametricReLU (ITensor &input, ITensor &slope) noexcept

Add a parametric ReLU layer to the network.

•

IConvolutionLayer ∗ addConvolutionNd (ITensor &input, int32 t nbOutputMaps, Dims kernelSize, Weights

kernelWeights, Weights biasWeights) noexcept

Add a multi-dimension convolution layer to the network.

•

IPoolingLayer ∗ addPoolingNd (ITensor &input, PoolingType type, Dims windowSize) noexcept

Add a multi-dimension pooling layer to the network.

•

IDeconvolutionLayer ∗ addDeconvolutionNd (ITensor &input, int32 t nbOutputMaps, Dims kernelSize, Weights

kernelWeights, Weights biasWeights) noexcept

Add a multi-dimension deconvolution layer to the network.

•

IScaleLayer ∗ addScaleNd (ITensor &input, ScaleMode mode, Weights shift, Weights scale, Weights power,

int32 t channelAxis) noexcept

Add a multi-dimension scale layer to the network.

•

IResizeLayer ∗ addResize (ITensor &input) noexcept

Add a resize layer to the network.

•

TRT DEPRECATED bool hasExplicitPrecision () const noexcept

True if network is an explicit precision network.

•

ILoop ∗ addLoop () noexcept

Add a loop to the network.

•

ISelectLayer ∗ addSelect (ITensor &condition, ITensor &thenInput, ITensor &elseInput) noexcept

Add a select layer to the network.

•

IAssertionLayer ∗ addAssertion (ITensor &condition, char const ∗message) noexcept

Add an assertion layer to the network.

•

IFillLayer ∗ addFill (Dims dimensions, FillOperation op) noexcept

Add a fill layer to the network.

•

TRT DEPRECATED IPaddingLayer ∗ addPaddingNd (ITensor &input, Dims prePadding, Dims postPadding)

noexcept

Add a padding layer to the network. Only 2D padding is currently supported.

•

bool setWeightsName (Weights weights, char const ∗name) noexcept

Associate a name with all current uses of the given weights.

•

void setErrorRecorder (IErrorRecorder ∗recorder) noexcept

Set the ErrorRecorder for this interface.

•

IErrorRecorder ∗ getErrorRecorder () const noexcept

get the ErrorRecorder assigned to this interface.

•

IDequantizeLayer ∗ addDequantize (ITensor &input, ITensor &scale) noexcept

Add a dequantization layer to the network.

•

IScatterLayer ∗ addScatter (ITensor &data, ITensor &indices, ITensor &updates, ScatterMode mode) noexcept

Add a Scatter layer to the network with specified mode and axis=0.

•

IQuantizeLayer ∗ addQuantize (ITensor &input, ITensor &scale) noexcept

Add a quantization layer to the network.

•

IIfConditional ∗ addIfConditional () noexcept

9.87 nvinfer1::INetworkDefinition Class Reference

417

Add an If-conditional layer to the network.

• IEinsumLayer ∗ addEinsum (ITensor ∗const ∗inputs, int32 t nbInputs, char const ∗equation) noexcept

Add an Einsum layer to the network.

• IGridSampleLayer ∗ addGridSample (ITensor &input, ITensor &grid) noexcept

Add a GridSample layer to the network.

• INMSLayer ∗ addNMS (ITensor &boxes, ITensor &scores, ITensor &maxOutputBoxesPerClass) noexcept

Add a non-maximum suppression layer to the network.

• IReverseSequenceLayer ∗ addReverseSequence (ITensor &input, ITensor &sequenceLens) noexcept

Add a ReverseSequence layer to the network.

• INormalizationLayer ∗ addNormalization (ITensor &input, ITensor &scale, ITensor &bias, uint32 t axesMask)

noexcept

Add a normalization layer to the network.

• virtual IBuilder & getBuilder () const noexcept

Return the builder from which this INetworkDefinition was created.

Protected Attributes

• apiv::VNetworkDefinition ∗ mImpl

Additional Inherited Members

9.87.1

Detailed Description

A network definition for input to the builder.

A network definition defines the structure of the network, and combined with a IBuilderConfig, is built into an engine

using an IBuilder. An INetworkDefinition can either have an implicit batch dimensions, specified at runtime, or all

dimensions explicit, full dims mode, in the network definition. The former mode, i.e. the implicit batch size mode, has

been deprecated. The function hasImplicitBatchDimension() can be used to query the mode of the network.

A network with implicit batch dimensions returns the dimensions of a layer without the implicit dimension, and instead

the batch is specified at execute/enqueue time. If the network has all dimensions specified, then the first dimension

follows elementwise broadcast rules: if it is 1 for some inputs and is some value N for all other inputs, then the first

dimension of each output is N, and the inputs with 1 for the first dimension are broadcast. Having divergent batch sizes

across inputs to a layer is not supported.

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.87.2

Constructor & Destructor Documentation

9.87.2.1

∼INetworkDefinition()

virtual nvinfer1::INetworkDefinition::∼INetworkDefinition ( ) [virtual], [default], [noexcept]

418

Class Documentation

9.87.3

Member Function Documentation

9.87.3.1

addActivation()

IActivationLayer ∗ nvinfer1::INetworkDefinition::addActivation (

ITensor & input,

ActivationType type ) [inline], [noexcept]

Add an activation layer to the network.

Parameters

input

The input tensor to the layer.

type

The type of activation function to apply.

Note that the setAlpha() and setBeta() methods must be used on the output for activations that require these parameters.

See also

IActivationLayer ActivationType

Warning

Int32 tensors are not valid input tensors.

Returns

The new activation layer, or nullptr if it could not be created.

9.87.3.2

addAssertion()

IAssertionLayer ∗ nvinfer1::INetworkDefinition::addAssertion (

ITensor & condition,

char const ∗ message ) [inline], [noexcept]

Add an assertion layer to the network.

Parameters

condition

The input tensor to the layer.

message

A message to print if the assertion fails.

9.87 nvinfer1::INetworkDefinition Class Reference

419

See also

IAssertionLayer

Returns

The new assertion layer, or nullptr if it could not be created.

The input tensor must be a boolean shape tensor.

9.87.3.3

addCast()

ICastLayer ∗ nvinfer1::INetworkDefinition::addCast (

ITensor & input,

DataType toType ) [inline], [noexcept]

Add a cast layer.

Parameters

input

The input tensor to the layer.

toType

The DataType of the output tensor

See also

ICastLayer

Returns

The new cast layer, or nullptr if it could not be created.

9.87.3.4

addConcatenation()

IConcatenationLayer ∗ nvinfer1::INetworkDefinition::addConcatenation (

ITensor ∗const ∗ inputs,

int32 t nbInputs ) [inline], [noexcept]

Add a concatenation layer to the network.

Parameters

inputs

The input tensors to the layer.

nbInputs

The number of input tensors.

420

Class Documentation

See also

IConcatenationLayer

Returns

The new concatenation layer, or nullptr if it could not be created.

Warning

All tensors must have the same dimensions except along the concatenation axis.

9.87.3.5

addConstant()

IConstantLayer ∗ nvinfer1::INetworkDefinition::addConstant (

Dims dimensions,

Weights weights ) [inline], [noexcept]

Add a constant layer to the network.

Parameters

dimensions

The dimensions of the constant.

weights

The constant value, represented as weights.

See also

IConstantLayer

Returns

The new constant layer, or nullptr if it could not be created.

If weights.type is DataType::kINT32, the output is a tensor of 32-bit indices. Otherwise the output is a tensor of real

values and the output type will be follow TensorRT's normal precision rules.

If tensors in the network have an implicit batch dimension, the constant is broadcast over that dimension.

If a wildcard dimension is used, the volume of the runtime dimensions must equal the number of weights specified.

Warning

DataType::kUINT8 not supported.

9.87 nvinfer1::INetworkDefinition Class Reference

421

9.87.3.6

addConvolution()

TRT DEPRECATED IConvolutionLayer ∗ nvinfer1::INetworkDefinition::addConvolution (

ITensor & input,

int32 t nbOutputMaps,

DimsHW kernelSize,

Weights kernelWeights,

Weights biasWeights ) [inline], [noexcept]

Add a convolution layer to the network.

Parameters

input

The input tensor to the convolution.

nbOutputMaps

The number of output feature maps for the convolution.

kernelSize

The HW-dimensions of the convolution kernel.

kernelWeights

The kernel weights for the convolution.

biasWeights

The bias weights for the convolution. Weights{} represents no bias.

See also

IConvolutionLayer

Warning

It is an error to specify a wildcard value for the 'C' dimension of the input tensor.

Int32 tensors are not valid input tensors.

Returns

The new convolution layer, or nullptr if it could not be created.

Deprecated Superseded by addConvolutionNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0

9.87.3.7

addConvolutionNd()

IConvolutionLayer ∗ nvinfer1::INetworkDefinition::addConvolutionNd (

ITensor & input,

int32 t nbOutputMaps,

Dims kernelSize,

Weights kernelWeights,

Weights biasWeights ) [inline], [noexcept]

Add a multi-dimension convolution layer to the network.

422

Class Documentation

Parameters

input

The input tensor to the convolution.

nbOutputMaps

The number of output feature maps for the convolution.

kernelSize

The multi-dimensions of the convolution kernel.

kernelWeights

The kernel weights for the convolution.

biasWeights

The bias weights for the convolution. Weights{} represents no bias.

See also

IConvolutionLayer

Warning

It is an error to specify a wildcard value for the 'C' dimension of the input tensor.

Int32 tensors are not valid input tensors.

Only 2D or 3D convolution is supported.

Returns

The new convolution layer, or nullptr if it could not be created.

9.87.3.8

addDeconvolution()

TRT DEPRECATED IDeconvolutionLayer ∗ nvinfer1::INetworkDefinition::addDeconvolution (

ITensor & input,

int32 t nbOutputMaps,

DimsHW kernelSize,

Weights kernelWeights,

Weights biasWeights ) [inline], [noexcept]

Add a deconvolution layer to the network.

Parameters

input

The input tensor to the layer.

nbOutputMaps

The number of output feature maps.

kernelSize

The HW-dimensions of the deconvolution kernel.

kernelWeights

The kernel weights for the deconvolution.

biasWeights

The bias weights for the deconvolution. Weights{} represents no bias.

9.87 nvinfer1::INetworkDefinition Class Reference

423

See also

IDeconvolutionLayer

Warning

It is an error to specify a wildcard value for the 'C' dimension of the input tensor.

Int32 tensors are not valid input tensors.

Returns

The new deconvolution layer, or nullptr if it could not be created.

Deprecated Superseded by addDeconvolutionNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0

9.87.3.9

addDeconvolutionNd()

IDeconvolutionLayer ∗ nvinfer1::INetworkDefinition::addDeconvolutionNd (

ITensor & input,

int32 t nbOutputMaps,

Dims kernelSize,

Weights kernelWeights,

Weights biasWeights ) [inline], [noexcept]

Add a multi-dimension deconvolution layer to the network.

Parameters

input

The input tensor to the layer.

nbOutputMaps

The number of output feature maps.

kernelSize

The multi-dimensions of the deconvolution kernel.

kernelWeights

The kernel weights for the deconvolution.

biasWeights

The bias weights for the deconvolution. Weights{} represents no bias.

See also

IDeconvolutionLayer

Warning

It is an error to specify a wildcard value for the 'C' dimension of the input tensor.

Int32 tensors are not valid input tensors.

Only 2D or 3D deconvolution is supported.

424

Class Documentation

Returns

The new deconvolution layer, or nullptr if it could not be created.

9.87.3.10 addDequantize()

IDequantizeLayer ∗ nvinfer1::INetworkDefinition::addDequantize (

ITensor & input,

ITensor & scale ) [inline], [noexcept]

Add a dequantization layer to the network.

Parameters

input

The input tensor to be quantized.

scale

A tensor with the scale value.

See also

IDequantizeLayer

input tensor data type must be DataType::kFLOAT. scale tensor data type must be DataType::kFLOAT. The

subgraph which terminates with the scale tensor must be a build-time constant.

Returns

The new quantization layer, or nullptr if it could not be created.

9.87.3.11 addEinsum()

IEinsumLayer ∗ nvinfer1::INetworkDefinition::addEinsum (

ITensor ∗const ∗ inputs,

int32 t nbInputs,

char const ∗ equation ) [inline], [noexcept]

Add an Einsum layer to the network.

Parameters

inputs

The input tensors to the layer.

nbInputs

The number of input tensors.

equation

The equation of the layer

9.87 nvinfer1::INetworkDefinition Class Reference

425

See also

IEinsumLayer

Returns

The new Einsum layer, or nullptr if it could not be created.

9.87.3.12 addElementWise()

IElementWiseLayer ∗ nvinfer1::INetworkDefinition::addElementWise (

ITensor & input1,

ITensor & input2,

ElementWiseOperation op ) [inline], [noexcept]

Add an elementwise layer to the network.

Parameters

input1

The first input tensor to the layer.

input2

The second input tensor to the layer.

The binary operation that the layer applies.

The input tensors must have the same rank and compatible type. Two types are compatible if they are the same type or

are both in the set {kFLOAT, kHALF}. For each dimension, their lengths must match, or one of them must be one. In

the latter case, the tensor is broadcast along that axis.

The output tensor has the same rank as the inputs. For each dimension, its length is the maximum of the lengths of the

corresponding input dimension.

The inputs are shape tensors if the output is a shape tensor.

See also

IElementWiseLayer

Returns

The new elementwise layer, or nullptr if it could not be created.

9.87.3.13 addFill()

IFillLayer ∗ nvinfer1::INetworkDefinition::addFill (

Dims dimensions,

FillOperation op ) [inline], [noexcept]

Add a fill layer to the network.

426

Class Documentation

Parameters

dimensions

The output tensor dimensions.

The fill operation that the layer applies.

Warning

For FillOperation::kLINSPACE, dimensions.nbDims must be 1.

This layer is non-deterministic across subsequent calls as the same inputs will produce different output tensors if op

is either FillOperation::kRANDOM UNIFORM or FillOperation::kRANDOM NORMAL due to random state being

shared across calls. The output tensors generated are determinstic when starting from the same initial state.

The network must not have an implicit batch dimension.

See also

IFillLayer

Returns

The new fill layer, or nullptr if it could not be created.

9.87.3.14 addFullyConnected()

TRT DEPRECATED IFullyConnectedLayer ∗ nvinfer1::INetworkDefinition::addFullyConnected (

ITensor & input,

int32 t nbOutputs,

Weights kernelWeights,

Weights biasWeights ) [inline], [noexcept]

Add a fully connected layer to the network.

Parameters

input

The input tensor to the layer.

nbOutputs

The number of outputs of the layer.

kernelWeights

The kernel weights for the fully connected layer.

biasWeights

The bias weights for the fully connected layer. Weights{} represents no bias.

See also

IFullyConnectedLayer

9.87 nvinfer1::INetworkDefinition Class Reference

427

Warning

It is an error to specify a wildcard value for the 'C' dimension of the input tensor.

Int32 tensors are not valid input tensors.

Returns

The new fully connected layer, or nullptr if it could not be created.

Deprecated Deprecated in TensorRT 8.4. Superseded by addMatrixMultiply().

9.87.3.15 addGather()

IGatherLayer ∗ nvinfer1::INetworkDefinition::addGather (

ITensor & data,

ITensor & indices,

int32 t axis ) [inline], [noexcept]

Add gather with mode GatherMode::kDEFAULT and specified axis and nbElementWiseDims=0.

Parameters

data

The tensor to gather values from.

indices

The tensor to get indices from to populate the output tensor.

axis

The axis in the data tensor to gather on.

See also

IGatherLayer

Returns

The new gather layer, or nullptr if it could not be created.

9.87.3.16 addGatherV2()

IGatherLayer ∗ nvinfer1::INetworkDefinition::addGatherV2 (

ITensor & data,

ITensor & indices,

GatherMode mode ) [inline], [noexcept]

Add gather with specified mode, axis=0 and nbElementWiseDims=0.

428

Class Documentation

Parameters

data

The tensor to gather values from.

indices

The tensor to get indices from to populate the output tensor.

mode

The gather mode.

See also

IGatherLayer

Returns

The new gather layer, or nullptr if it could not be created.

9.87.3.17 addGridSample()

IGridSampleLayer ∗ nvinfer1::INetworkDefinition::addGridSample (

ITensor & input,

ITensor & grid ) [inline], [noexcept]

Add a GridSample layer to the network.

Parameters

input

The input tensor to the layer.

grid

The grid tensor to the layer.

See also

IGridSampleLayer

Creates a GridSample layer with a InterpolationMode::kLINEAR, unaligned corners, and SampleMode::kFILL for

4d-shape input tensors.

Returns

The new GridSample layer, or nullptr if it could not be created.

9.87.3.18 addIdentity()

IIdentityLayer ∗ nvinfer1::INetworkDefinition::addIdentity (

ITensor & input ) [inline], [noexcept]

Add an identity layer.

9.87 nvinfer1::INetworkDefinition Class Reference

429

Parameters

input

The input tensor to the layer.

See also

IIdentityLayer

Returns

The new identity layer, or nullptr if it could not be created.

9.87.3.19 addIfConditional()

IIfConditional ∗ nvinfer1::INetworkDefinition::addIfConditional ( ) [inline], [noexcept]

Add an If-conditional layer to the network.

An IIfConditional provides a way to conditionally execute parts of the network.

See also

IIfConditional

Returns

The new conditional layer, or nullptr if network has an implicit batch dimension or this version of TensorRT does

not support conditional execution.

9.87.3.20 addInput()

ITensor ∗ nvinfer1::INetworkDefinition::addInput (

char const ∗ name,

DataType type,

Dims dimensions ) [inline], [noexcept]

Add an input tensor to the network.

The name of the input tensor is used to find the index into the buffer array for an engine built from the network. The

volume must be less than 2^∧31 elements.

For networks with an implicit batch dimension, this volume includes the batch dimension with its length set to the

maximum batch size. For networks with all explicit dimensions and with wildcard dimensions, the volume is based

on the maxima specified by an IOptimizationProfile.Dimensions are normally non-negative integers. The excep-

tion is that in networks with all explicit dimensions, -1 can be used as a wildcard for a dimension to be specified

at runtime. Input tensors with such a wildcard must have a corresponding entry in the IOptimizationProfiles indi-

cating the permitted extrema, and the input dimensions must be set by IExecutionContext::setBindingDimensions.

Different IExecutionContext instances can have different dimensions. Wildcard dimensions are only supported for

EngineCapability::kSTANDARD. They are not supported in safety contexts. DLA does not support Wildcard dimen-

sions.

Tensor dimensions are specified independent of format. For example, if a tensor is formatted in ”NHWC” or a vec-

torized format, the dimensions are still specified in the order{N, C, H, W}. For 2D images with a channel dimension,

the last three dimensions are always {C,H,W}. For 3D images with a channel dimension, the last four dimensions are

always {C,D,H,W}.

430

Class Documentation

Parameters

name

The name of the tensor.

type

The type of the data held in the tensor.

dimensions

The dimensions of the tensor.

Warning

It is an error to specify a wildcard value on a dimension that is determined by trained parameters.

If run on DLA with explicit dimensions, only leading dimension can be a wildcard. And provided profile

must have same minimum, optimum, and maximum dimensions.

The string name must be null-terminated, and be at most 4096 bytes including the terminator.

See also

ITensor

Returns

The new tensor or nullptr if there is an error.

9.87.3.21 addLoop()

ILoop ∗ nvinfer1::INetworkDefinition::addLoop ( ) [inline], [noexcept]

Add a loop to the network.

An ILoop provides a way to specify a recurrent subgraph.

Returns

Pointer to ILoop that can be used to add loop boundary layers for the loop, or nullptr if network has an implicit

batch dimension or this version of TensorRT does not support loops.

The network must not have an implicit batch dimension.

9.87.3.22 addLRN()

ILRNLayer ∗ nvinfer1::INetworkDefinition::addLRN (

ITensor & input,

int32 t window,

float alpha,

float beta,

float k ) [inline], [noexcept]

Add a LRN layer to the network.

9.87 nvinfer1::INetworkDefinition Class Reference

431

Parameters

input

The input tensor to the layer.

window

The size of the window.

alpha

The alpha value for the LRN computation.

beta

The beta value for the LRN computation.

The k value for the LRN computation.

See also

ILRNLayer

Warning

Int32 tensors are not valid input tensors.

Returns

The new LRN layer, or nullptr if it could not be created.

9.87.3.23 addMatrixMultiply()

IMatrixMultiplyLayer ∗ nvinfer1::INetworkDefinition::addMatrixMultiply (

ITensor & input0,

MatrixOperation op0,

ITensor & input1,

MatrixOperation op1 ) [inline], [noexcept]

Add a MatrixMultiply layer to the network.

Parameters

input0

The first input tensor (commonly A).

op0

The operation to apply to input0.

input1

The second input tensor (commonly B).

op1

The operation to apply to input1.

The inputs are shape tensors if the output is a shape tensor.

See also

IMatrixMultiplyLayer

432

Class Documentation

Warning

Int32 tensors are not valid input tensors.

Returns

The new matrix multiply layer, or nullptr if it could not be created.

9.87.3.24 addNMS()

INMSLayer ∗ nvinfer1::INetworkDefinition::addNMS (

ITensor & boxes,

ITensor & scores,

ITensor & maxOutputBoxesPerClass ) [inline], [noexcept]

Add a non-maximum suppression layer to the network.

Parameters

boxes

The input boxes tensor to the layer.

scores

The input scores tensor to the layer.

maxOutputBoxesPerClass

The input maxOutputBoxesPerClass tensor to the layer.

See also

INMSLayer

Returns

The new NMS layer, or nullptr if it could not be created.

9.87.3.25 addNonZero()

INonZeroLayer ∗ nvinfer1::INetworkDefinition::addNonZero (

ITensor & input ) [inline], [noexcept]

Add a nonzero layer to the network.

Parameters

input

The input tensor to the layer.

9.87 nvinfer1::INetworkDefinition Class Reference

433

See also

INonZeroLayer

Returns

The new nonzero layer, or nullptr if it could be created.

9.87.3.26 addNormalization()

INormalizationLayer ∗ nvinfer1::INetworkDefinition::addNormalization (

ITensor & input,

ITensor & scale,

ITensor & bias,

uint32 t axesMask ) [inline], [noexcept]

Add a normalization layer to the network.

Parameters

input

The input tensor to the layer.

scale

The scale tensor used to scale the normalized output.

bias

The bias tensor used to scale the normalized output.

axesMask

The axes on which to perform mean calculations. The bit in position i of bitmask axesMask

corresponds to explicit dimension i of the result. E.g., the least significant bit corresponds to the first

explicit dimension and the next to least significant bit corresponds to the second explicit dimension.

The normalization layer works by performing normalization of the tensor input on the specified axesMask. The

result is then scaled by multiplying with scale and adding bias.

The shape of scale and bias are expected the be the same, and must have the same rank and be unidirectionally

broadcastable to the shape of input.

See also

INormalizationLayer

Returns

The new normalization layer, or nullptr if it could not be created.

434

Class Documentation

9.87.3.27 addOneHot()

IOneHotLayer ∗ nvinfer1::INetworkDefinition::addOneHot (

ITensor & indices,

ITensor & values,

ITensor & depth,

int32 t axis ) [inline], [noexcept]

Add a OneHot layer to the network.

Parameters

indices

- tensor containing indices where on value should be set.

values

- a 2-element tensor, consisting of [off value, on value].

depth

- tensor containing the width of the added one-hot dimension.

axis

- the axis to add the one-hot encoding to.

See also

IOneHotLayer

Returns

The new OneHot layer, or nullptr if it could not be created.

9.87.3.28 addPadding()

TRT DEPRECATED IPaddingLayer ∗ nvinfer1::INetworkDefinition::addPadding (

ITensor & input,

DimsHW prePadding,

DimsHW postPadding ) [inline], [noexcept]

Add a padding layer to the network.

Parameters

input

The input tensor to the layer.

prePadding

The padding to apply to the start of the tensor.

postPadding

The padding to apply to the end of the tensor.

See also

IPaddingLayer

9.87 nvinfer1::INetworkDefinition Class Reference

435

Returns

The new padding layer, or nullptr if it could not be created.

Deprecated Superseded by addPaddingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0

9.87.3.29 addPaddingNd()

TRT DEPRECATED IPaddingLayer ∗ nvinfer1::INetworkDefinition::addPaddingNd (

ITensor & input,

Dims prePadding,

Dims postPadding ) [inline], [noexcept]

Add a padding layer to the network. Only 2D padding is currently supported.

Parameters

input

The input tensor to the layer.

prePadding

The padding to apply to the start of the tensor.

postPadding

The padding to apply to the end of the tensor.

See also

IPaddingLayer

Returns

The new padding layer, or nullptr if it could not be created.

Deprecated Deprecated in TensorRT 8.0. Superseded by addSlice().

9.87.3.30 addParametricReLU()

IParametricReLULayer ∗ nvinfer1::INetworkDefinition::addParametricReLU (

ITensor & input,

ITensor & slope ) [inline], [noexcept]

Add a parametric ReLU layer to the network.

436

Class Documentation

Parameters

input

The input tensor to the layer.

slope

The slope tensor to the layer. This tensor should be unidirectionally broadcastable to the input tensor.

See also

IParametricReLULayer

Warning

Int32 tensors are not valid input tensors.

Returns

The new parametric ReLU layer, or nullptr if it could not be created.

9.87.3.31 addPluginV2()

IPluginV2Layer ∗ nvinfer1::INetworkDefinition::addPluginV2 (

ITensor ∗const ∗ inputs,

int32 t nbInputs,

IPluginV2 & plugin ) [inline], [noexcept]

Add a plugin layer to the network using the IPluginV2 interface.

Parameters

inputs

The input tensors to the layer.

nbInputs

The number of input tensors.

plugin

The layer plugin.

See also

IPluginV2Layer

Warning

Dimension wildcard are only supported with IPluginV2DynamicExt or IPluginV2IOExt plugins.

Int32 tensors are not valid input tensors.

Returns

The new plugin layer, or nullptr if it could not be created.

9.87 nvinfer1::INetworkDefinition Class Reference

437

9.87.3.32 addPooling()

TRT DEPRECATED IPoolingLayer ∗ nvinfer1::INetworkDefinition::addPooling (

ITensor & input,

PoolingType type,

DimsHW windowSize ) [inline], [noexcept]

Add a pooling layer to the network.

Parameters

input

The input tensor to the layer.

type

The type of pooling to apply.

windowSize

The size of the pooling window.

See also

IPoolingLayer PoolingType

Warning

Int32 tensors are not valid input tensors.

Returns

The new pooling layer, or nullptr if it could not be created.

Deprecated Superseded by addPoolingNd. Deprecated prior to TensorRT 8.0 and will be removed in 9.0

9.87.3.33 addPoolingNd()

IPoolingLayer ∗ nvinfer1::INetworkDefinition::addPoolingNd (

ITensor & input,

PoolingType type,

Dims windowSize ) [inline], [noexcept]

Add a multi-dimension pooling layer to the network.

Parameters

input

The input tensor to the layer.

type

The type of pooling to apply.

windowSize

The size of the pooling window.

438

Class Documentation

See also

IPoolingLayer PoolingType

Warning

Int32 tensors are not valid input tensors.

Only 2D or 3D pooling is supported.

Returns

The new pooling layer, or nullptr if it could not be created.

9.87.3.34 addQuantize()

IQuantizeLayer ∗ nvinfer1::INetworkDefinition::addQuantize (

ITensor & input,

ITensor & scale ) [inline], [noexcept]

Add a quantization layer to the network.

Parameters

input

The input tensor to be quantized.

scale

A tensor with the scale value.

See also

IQuantizeLayer

input tensor data type must be DataType::kFLOAT. scale tensor data type must be DataType::kFLOAT. The

subgraph which terminates with the scale tensor must be a build-time constant.

Returns

The new quantization layer, or nullptr if it could not be created.

9.87.3.35 addRaggedSoftMax()

IRaggedSoftMaxLayer ∗ nvinfer1::INetworkDefinition::addRaggedSoftMax (

ITensor & input,

ITensor & bounds ) [inline], [noexcept]

Add a RaggedSoftMax layer to the network.

9.87 nvinfer1::INetworkDefinition Class Reference

439

Parameters

input

The ZxS input tensor.

bounds

The Zx1 bounds tensor.

See also

IRaggedSoftMaxLayer

Warning

The bounds tensor cannot have the last dimension be the wildcard character.

Int32 tensors are not valid input tensors.

Returns

The new RaggedSoftMax layer, or nullptr if it could not be created.

9.87.3.36 addReduce()

IReduceLayer ∗ nvinfer1::INetworkDefinition::addReduce (

ITensor & input,

ReduceOperation operation,

uint32 t reduceAxes,

bool keepDimensions ) [inline], [noexcept]

Add a reduce layer to the network.

Parameters

input

The input tensor to the layer.

operation

The reduction operation to perform.

reduceAxes

The reduction dimensions. The bit in position i of bitmask reduceAxes corresponds to explicit

dimension i if result. E.g., the least significant bit corresponds to the first explicit dimension

and the next to least significant bit corresponds to the second explicit dimension.

keepDimensions

The boolean that specifies whether or not to keep the reduced dimensions in the output of the

layer.

The reduce layer works by performing an operation specified by operation to reduce the tensor input across the

axes specified by reduceAxes.

See also

IReduceLayer

440

Class Documentation

Warning

If output is an Int32 shape tensor, ReduceOperation::kAVG is unsupported.

Returns

The new reduce layer, or nullptr if it could not be created.

9.87.3.37 addResize()

IResizeLayer ∗ nvinfer1::INetworkDefinition::addResize (

ITensor & input ) [inline], [noexcept]

Add a resize layer to the network.

Parameters

input

The input tensor to the layer.

See also

IResizeLayer

Warning

Int32 tensors are not valid input tensors.

Returns

The new resize layer, or nullptr if it could not be created.

9.87.3.38 addReverseSequence()

IReverseSequenceLayer ∗ nvinfer1::INetworkDefinition::addReverseSequence (

ITensor & input,

ITensor & sequenceLens ) [inline], [noexcept]

Add a ReverseSequence layer to the network.

9.87 nvinfer1::INetworkDefinition Class Reference

441

Parameters

input

The input tensor to the layer. Must have rank >= 2.

sequenceLens

1D tensor specifying lengths of sequences to reverse in a batch. The length of the sequenceLens

tensor must be equal to the size of the dimension in input tensor specified by batchAxis.

See also

IReverseSequenceLayer

Returns

The new ReverseSequence layer, or nullptr if it could not be created.

9.87.3.39 addRNNv2()

TRT DEPRECATED IRNNv2Layer ∗ nvinfer1::INetworkDefinition::addRNNv2 (

ITensor & input,

int32 t layerCount,

int32 t hiddenSize,

int32 t maxSeqLen,

RNNOperation op ) [inline], [noexcept]

Add an layerCount deep RNN layer to the network with hiddenSize internal states that can take a batch with

fixed or variable sequence lengths.

Parameters

input

The input tensor to the layer (see below).

layerCount

The number of layers in the RNN.

hiddenSize

Size of the internal hidden state for each layer.

maxSeqLen

Maximum sequence length for the input.

The type of RNN to execute.

By default, the layer is configured with RNNDirection::kUNIDIRECTION and RNNInputMode::kLINEAR. To

change these settings, use IRNNv2Layer::setDirection() and IRNNv2Layer::setInputMode().

Weights and biases for the added layer should be set using IRNNv2Layer::setWeightsForGate() and IRNNv2Layer::setBiasForGate()

prior to building an engine using this network.

The input tensors must be of the type DataType::kFLOAT or DataType::kHALF. The layout of the weights is row

major and must be the same datatype as the input tensor. weights contain 8 matrices and bias contains 8 vectors.

See IRNNv2Layer::setWeightsForGate() and IRNNv2Layer::setBiasForGate() for details on the required input format

for weights and bias.

442

Class Documentation

The input ITensor should contain zero or more index dimensions {N1, ..., Np}, followed by two dimensions,

defined as follows:

• S max is the maximum allowed sequence length (number of RNN iterations)

• E specifies the embedding length (unless RNNInputMode::kSKIP is set, in which case it should match get←↩

HiddenSize()).

By default, all sequences in the input are assumed to be size maxSeqLen. To provide explicit sequence lengths for

each input sequence in the batch, use IRNNv2Layer::setSequenceLengths().

The RNN layer outputs up to three tensors.

The first output tensor is the output of the final RNN layer across all timesteps, with dimensions {N1, ..., Np,

S max, H}:

• N1..Np are the index dimensions specified by the input tensor

• S max is the maximum allowed sequence length (number of RNN iterations)

• H is an output hidden state (equal to getHiddenSize() or 2x getHiddenSize())

The second tensor is the final hidden state of the RNN across all layers, and if the RNN is an LSTM (i.e. getOperation()

is RNNOperation::kLSTM), then the third tensor is the final cell state of the RNN across all layers. Both the second

and third output tensors have dimensions {N1, ..., Np, L, H}:

• N1..Np are the index dimensions specified by the input tensor

• ListhenumberoflayersintheRNN,equaltogetLayerCount()ifgetDirectionisRNNDirection::kUNIDIRECTION,

and 2x getLayerCount() if getDirection is RNNDirection::kBIDIRECTION. In the bi-directional case, layer l's

final forward hidden state is stored in L = 2∗l, and final backward hidden state is stored in L= 2∗l + 1.

• H is the hidden state for each layer, equal to getHiddenSize().

See also

IRNNv2Layer

Deprecated Deprecated prior to TensorRT 8.0 and will be removed in 9.0. Superseded by INetworkDefinition::addLoop().

Warning

RNN inputs do not support wildcard dimensions or explicit batch size networks.

Int32 tensors are not valid input tensors, only for sequence lengths.

Returns

The new RNN layer, or nullptr if it could not be created.

9.87 nvinfer1::INetworkDefinition Class Reference

443

9.87.3.40 addScale()

IScaleLayer ∗ nvinfer1::INetworkDefinition::addScale (

ITensor & input,

ScaleMode mode,

Weights shift,

Weights scale,

Weights power ) [inline], [noexcept]

Add a Scale layer to the network.

Parameters

input

The input tensor to the layer. This tensor is required to have a minimum of 3 dimensions in implicit batch

mode and a minimum of 4 dimensions in explicit batch mode.

mode

The scaling mode.

shift

The shift value.

scale

The scale value.

power

The power value.

If the weights are available, then the size of weights are dependent on the ScaleMode. For ScaleMode::kUNIFORM,

the number of weights equals 1. For ScaleMode::kCHANNEL, the number of weights equals the channel dimension.

For ScaleMode::kELEMENTWISE, the number of weights equals the product of the last three dimensions of the input.

See also

addScaleNd

IScaleLayer

Warning

Int32 tensors are not valid input tensors.

Returns

The new Scale layer, or nullptr if it could not be created.

9.87.3.41 addScaleNd()

IScaleLayer ∗ nvinfer1::INetworkDefinition::addScaleNd (

ITensor & input,

ScaleMode mode,

Weights shift,

Weights scale,

Weights power,

int32 t channelAxis ) [inline], [noexcept]

Add a multi-dimension scale layer to the network.

444

Class Documentation

Parameters

input

The input tensor to the layer.

mode

The scaling mode.

shift

The shift value.

scale

The scale value.

power

The power value.

channelAxis

The channel axis.

If the weights are available, then the size of weights are dependent on the ScaleMode. For ScaleMode::kUNIFORM,

the number of weights equals 1. For ScaleMode::kCHANNEL, the number of weights equals the channel dimension.

For ScaleMode::kELEMENTWISE, the number of weights equals the product of all input dimensions at channelAxis

and beyond.

For example, if the inputs dimensions are [A,B,C,D,E,F], and channelAxis=2: For ScaleMode::kUNIFORM,

the number of weights is equal to

For ScaleMode::kCHANNEL, the number of weights is C. For

ScaleMode::kELEMENTWISE, the number of weights is C∗D∗E∗F.

channelAxis can also be set explicitly using setChannelAxis().

See also

IScaleLayer

setChannelAxis()

Warning

Int32 tensors are not valid input tensors.

Only 2D or 3D scale is supported.

Returns

The new Scale layer, or nullptr if it could not be created.

9.87.3.42 addScatter()

IScatterLayer ∗ nvinfer1::INetworkDefinition::addScatter (

ITensor & data,

ITensor & indices,

ITensor & updates,

ScatterMode mode ) [inline], [noexcept]

Add a Scatter layer to the network with specified mode and axis=0.

9.87 nvinfer1::INetworkDefinition Class Reference

445

Parameters

data

The input tensor to be updated with additional values.

indices

indices of the elements to be updated.

updates

values to be used for updates.

mode

scatter mode.

See also

IScatterLayer

indices tensor data type must be DataType::kINT32. updates tensor data type must be the same as data

Returns

The new Scatter layer, or nullptr if it could not be created.

9.87.3.43 addSelect()

ISelectLayer ∗ nvinfer1::INetworkDefinition::addSelect (

ITensor & condition,

ITensor & thenInput,

ITensor & elseInput ) [inline], [noexcept]

Add a select layer to the network.

Parameters

condition

The condition tensor to the layer. Must have type DataType::kBOOL.

thenInput

The ”then” input tensor to the layer.

elseInput

The ”else” input tensor to the layer.

All three input tensors must have the same rank, and along each axis must have the same length or a length of one. If

the length is one, the tensor is broadcast along that axis. The output tensor has the dimensions of the inputs AFTER

the broadcast rule is applied. For example, given:

dimensions of condition: [1,1,5,9] dimensions of thenInput: [1,1,5,9] dimensions of elseInput: [1,3,1,9]

the output dimensions are [1,3,5,9], and the output contents are defined by:

output[0,i,j,k] = condition[0,0,j,k] ? thenInput[0,0,j,k] : elseInput[0,i,0,k]

The output dimensions are not necessarily the max of the input dimensions if any input is an empty tensor. For example,

if in the preceding example, 5 is changed to 0:

446

Class Documentation

dimensions of condition: [1,1,0,9] dimensions of thenInput: [1,1,0,9] dimensions of elseInput: [1,3,1,9]

then the output dimensions are [1,3,0,9].

The network must not have an implicit batch dimension.

The inputs are shape tensors if the output is a shape tensor.

See also

ISelectLayer

Returns

The new select layer, or nullptr if it could not be created.

9.87.3.44 addShape()

IShapeLayer ∗ nvinfer1::INetworkDefinition::addShape (

ITensor & input ) [inline], [noexcept]

Add a shape layer to the network.

Parameters

input

The input tensor to the layer.

See also

IShapeLayer

Warning

addShape is only supported when hasImplicitBatchDimensions is false.

Returns

The new shape layer, or nullptr if it could not be created.

9.87.3.45 addShuffle()

IShuffleLayer ∗ nvinfer1::INetworkDefinition::addShuffle (

ITensor & input ) [inline], [noexcept]

Add a shuffle layer to the network.

9.87 nvinfer1::INetworkDefinition Class Reference

447

Parameters

input

The input tensor to the layer.

See also

IShuffleLayer

Returns

The new shuffle layer, or nullptr if it could not be created.

9.87.3.46 addSlice()

ISliceLayer ∗ nvinfer1::INetworkDefinition::addSlice (

ITensor & input,

Dims start,

Dims size,

Dims stride ) [inline], [noexcept]

Add a slice layer to the network.

Parameters

input

The input tensor to the layer.

start

The start offset

size

The output dimension

stride

The slicing stride

Positive, negative, zero stride values, and combinations of them in different dimensions are allowed.

See also

ISliceLayer

Returns

The new slice layer, or nullptr if it could not be created.

448

Class Documentation

9.87.3.47 addSoftMax()

ISoftMaxLayer ∗ nvinfer1::INetworkDefinition::addSoftMax (

ITensor & input ) [inline], [noexcept]

Add a SoftMax layer to the network.

See also

ISoftMaxLayer

Warning

Int32 tensors are not valid input tensors.

Returns

The new SoftMax layer, or nullptr if it could not be created.

9.87.3.48 addTopK()

ITopKLayer ∗ nvinfer1::INetworkDefinition::addTopK (

ITensor & input,

TopKOperation op,

int32 t k,

uint32 t reduceAxes ) [inline], [noexcept]

Add a TopK layer to the network.

The TopK layer has two outputs of the same dimensions. The first contains data values, the second contains index

positions for the values. Output values are sorted, largest first for operation kMAX and smallest first for operation

kMIN.

Currently only values of K up to 3840 are supported.

Parameters

input

The input tensor to the layer.

Operation to perform.

The number of elements to keep. For dynamic k, use the setInput() method to pass in k as a tensor

instead, which will override the static k value passed here in calculations.

reduceAxes

The reduction dimensions. The bit in position i of bitmask reduceAxes corresponds to explicit

dimension i of the result. E.g., the least significant bit corresponds to the first explicit dimension and

the next to least significant bit corresponds to the second explicit dimension.

9.87 nvinfer1::INetworkDefinition Class Reference

449

Currently reduceAxes must specify exactly one dimension, and it must be one of the last four dimensions.

See also

ITopKLayer

Warning

Int32 tensors are not valid input tensors.

Returns

The new TopK layer, or nullptr if it could not be created.

9.87.3.49 addUnary()

IUnaryLayer ∗ nvinfer1::INetworkDefinition::addUnary (

ITensor & input,

UnaryOperation operation ) [inline], [noexcept]

Add a unary layer to the network.

Parameters

input

The input tensor to the layer.

operation

The operation to apply.

See also

IUnaryLayer

Generally the input must have a floating-point type (or kINT8 as a quantized float), except for the following

operations:

• kSIGN accepts a floating-point or Int32 tensor.

• kNOT requires a Bool tensor.

The input is a shape tensor if the output is a shape tensor.

Returns

The new unary layer, or nullptr if it could not be created

450

Class Documentation

9.87.3.50 destroy()

TRT DEPRECATED void nvinfer1::INetworkDefinition::destroy ( ) [inline], [noexcept]

Destroy this INetworkDefinition object.

Deprecated Deprecated in TensorRT 8.0. Superseded by delete.

Warning

Calling destroy on a managed pointer will result in a double-free error.

9.87.3.51 getBuilder()

virtual IBuilder & nvinfer1::INetworkDefinition::getBuilder ( ) const [inline], [virtual], [noexcept]

Return the builder from which this INetworkDefinition was created.

See also

IBuilder::createNetworkV2

Returns

the builder

9.87.3.52 getErrorRecorder()

IErrorRecorder ∗ nvinfer1::INetworkDefinition::getErrorRecorder ( ) const [inline], [noexcept]

get the ErrorRecorder assigned to this interface.

Retrieves the assigned error recorder object for the given class. A nullptr will be returned if setErrorRecorder has not

been called.

Returns

A pointer to the IErrorRecorder object that has been registered.

See also

setErrorRecorder()

9.87.3.53 getInput()

ITensor ∗ nvinfer1::INetworkDefinition::getInput (

int32 t index ) const [inline], [noexcept]

Get the input tensor specified by the given index.

9.87 nvinfer1::INetworkDefinition Class Reference

451

Parameters

index

The index of the input tensor.

Returns

The input tensor, or nullptr if the index is out of range.

Note

adding inputs invalidates indexing here

See also

getNbInputs()

9.87.3.54 getLayer()

ILayer ∗ nvinfer1::INetworkDefinition::getLayer (

int32 t index ) const [inline], [noexcept]

Get the layer specified by the given index.

Parameters

index

The index of the layer.

Returns

The layer, or nullptr if the index is out of range.

See also

getNbLayers()

9.87.3.55 getName()

char const ∗ nvinfer1::INetworkDefinition::getName ( ) const [inline], [noexcept]

Returns the name associated with the network.

The memory pointed to by getName() is owned by the INetworkDefinition object.

9.87 nvinfer1::INetworkDefinition Class Reference

453

9.87.3.58 getNbOutputs()

int32 t nvinfer1::INetworkDefinition::getNbOutputs ( ) const [inline], [noexcept]

Get the number of outputs in the network.

The outputs include those marked by markOutput or markOutputForShapes.

Returns

The number of outputs in the network.

See also

getOutput()

9.87.3.59 getOutput()

ITensor ∗ nvinfer1::INetworkDefinition::getOutput (

int32 t index ) const [inline], [noexcept]

Get the output tensor specified by the given index.

Parameters

index

The index of the output tensor.

Returns

The output tensor, or nullptr if the index is out of range.

Note

adding inputs invalidates indexing here

See also

getNbOutputs()

454

Class Documentation

9.87.3.60 hasExplicitPrecision()

TRT DEPRECATED bool nvinfer1::INetworkDefinition::hasExplicitPrecision ( ) const [inline], [noexcept]

True if network is an explicit precision network.

Deprecated Deprecated in TensorRT 8.0.

See also

createNetworkV2

Returns

True if network has explicit precision, false otherwise.

9.87.3.61 hasImplicitBatchDimension()

bool nvinfer1::INetworkDefinition::hasImplicitBatchDimension ( ) const [inline], [noexcept]

Query whether the network was created with an implicit batch dimension.

Returns

True if tensors have implicit batch dimension, false otherwise.

This is a network-wide property. Either all tensors in the network have an implicit batch dimension or none of them

do.

hasImplicitBatchDimension() is true if and only if this INetworkDefinition was created with createNetworkV2() with-

out NetworkDefinitionCreationFlag::kEXPLICIT BATCH flag.

See also

createNetworkV2

9.87.3.62 markOutput()

void nvinfer1::INetworkDefinition::markOutput (

ITensor & tensor ) [inline], [noexcept]

Mark a tensor as a network output.

9.87 nvinfer1::INetworkDefinition Class Reference

455

Parameters

tensor

The tensor to mark as an output tensor.

Warning

It is an error to mark a network input as an output.

It is an error to mark a tensor inside an ILoop or an IIfConditional as an output.

9.87.3.63 markOutputForShapes()

bool nvinfer1::INetworkDefinition::markOutputForShapes (

ITensor & tensor ) [inline], [noexcept]

Enable tensor's value to be computed by IExecutionContext::getShapeBinding.

Returns

True if successful, false if tensor is already marked as an output.

The tensor must be of type DataType::kINT32 and have no more than one dimension.

Warning

The tensor must have dimensions that can be determined to be constants at build time.

It is an error to mark a network input as a shape output.

See also

isShapeBinding(), getShapeBinding()

9.87.3.64 removeTensor()

void nvinfer1::INetworkDefinition::removeTensor (

ITensor & tensor ) [inline], [noexcept]

remove a tensor from the network definition.

Parameters

tensor

the tensor to remove

456

Class Documentation

It is illegal to remove a tensor that is the input or output of a layer. if this method is called with such a tensor, a

warning will be emitted on the log and the call will be ignored. Its intended use is to remove detached tensors after e.g.

concatenating two networks with Layer::setInput().

9.87.3.65 setErrorRecorder()

void nvinfer1::INetworkDefinition::setErrorRecorder (

IErrorRecorder ∗ recorder ) [inline], [noexcept]

Set the ErrorRecorder for this interface.

Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function

will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder

with the interface, resulting in a call to decRefCount if a recorder has been registered.

If an error recorder is not set, messages will be sent to the global log stream.

Parameters

recorder

The error recorder to register with this interface.

See also

getErrorRecorder()

9.87.3.66 setName()

void nvinfer1::INetworkDefinition::setName (

char const ∗ name ) [inline], [noexcept]

Sets the name of the network.

Parameters

name

The name to assign to this network.

Set the name of the network so that it can be associated with a built engine. The name must be a null-terminated

C-style string. TensorRT makes no use of this string except storing it as part of the engine so that it may be retrieved at

runtime. A name unique to the builder will be generated by default.

This method copies the name string.

9.87 nvinfer1::INetworkDefinition Class Reference

457

Warning

The string name must be null-terminated, and be at most 4096 bytes including the terminator.

See also

INetworkDefinition::getName(), ISafeCudaEngine::getName()

Returns

none

9.87.3.67 setWeightsName()

bool nvinfer1::INetworkDefinition::setWeightsName (

Weights weights,

char const ∗ name ) [inline], [noexcept]

Associate a name with all current uses of the given weights.

The name must be set after the Weights are used in the network. Lookup is associative. The name applies to all

Weights with matching type, value pointer, and count. If Weights with a matching value pointer, but different type or

count exists in the network, an error message is issued, the name is rejected, and return false. If the name has already

been used for other weights, return false. A nullptr causes the weights to become unnamed, i.e. clears any previous

name.

Parameters

weights

The weights to be named.

name

The name to associate with the weights.

Returns

true on success.

Warning

The string name must be null-terminated, and be at most 4096 bytes including the terminator.

9.87.3.68 unmarkOutput()

void nvinfer1::INetworkDefinition::unmarkOutput (

ITensor & tensor ) [inline], [noexcept]

unmark a tensor as a network output.

458

Class Documentation

Parameters

tensor

The tensor to unmark as an output tensor.

see markOutput()

9.87.3.69 unmarkOutputForShapes()

bool nvinfer1::INetworkDefinition::unmarkOutputForShapes (

ITensor & tensor ) [inline], [noexcept]

Undo markOutputForShapes.

Warning

inputs to addShape cannot contain wildcard dimension values.

Returns

True if successful, false if tensor is not marked as an output.

9.87.4

Member Data Documentation

9.87.4.1

mImpl

apiv::VNetworkDefinition∗ nvinfer1::INetworkDefinition::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.88

nvinfer1::INMSLayer Class Reference

A non-maximum suppression layer in a network definition.

#include <NvInfer.h>

Inheritance diagram for nvinfer1::INMSLayer:

nvinfer1::INoCopy

nvinfer1::ILayer

nvinfer1::INMSLayer

9.88 nvinfer1::INMSLayer Class Reference

459

Public Member Functions

• void setBoundingBoxFormat (BoundingBoxFormat fmt) noexcept

Set the bounding box format parameter for the layer.

• BoundingBoxFormat getBoundingBoxFormat () const noexcept

Get the bounding box format parameter for the layer.

• void setTopKBoxLimit (int32 t limit) noexcept

Set the TopK box limit parameter for the layer.

• int32 t getTopKBoxLimit () const noexcept

Get the TopK box limit parameter for the layer.

• void setInput (int32 t index, ITensor &tensor) noexcept

Append or replace an input of this layer with a specific tensor.

Protected Member Functions

• virtual ∼INMSLayer () noexcept=default

Protected Attributes

• apiv::VNMSLayer ∗ mImpl

9.88.1

Detailed Description

A non-maximum suppression layer in a network definition.

The NMS algorithm iterates through a set of bounding boxes and their confidence scores, in decreasing order of score.

Boxes are selected if their score is above a given threshold, and their intersection-over-union (IoU) with previously

selected boxes is less than or equal to a given threshold. This layer implements NMS per batch item and per class.

For each batch item, the ordering of candidate bounding boxes with the same score is unspecified.

The layer has the following inputs, in order of input index:

• Boxes contains the input bounding boxes. It is a linear tensor of type kFLOAT or kHALF. It has shape [batchSize,

numInputBoundingBoxes, numClasses, 4] if the boxes are per class, or [batchSize, numInputBoundingBoxes, 4]

if the same boxes are to be used for each class.

• Scores contains the per-box scores. It is a linear tensor of the same type as Boxes. It has shape [batchSize,

numInputBoundingBoxes, numClasses].

• MaxOutputBoxesPerClass is the maximum number of output boxes per batch item per class. It is a scalar (0D

tensor) of type kINT32.

• IoUThreshold is the maximum IoU for selected boxes. It is a scalar (0D tensor) of type kFLOAT in the range

[0.0f, 1.0f]. It is an optional input with default 0.0f.

• ScoreThreshold is the value that a box score must exceed in order to be selected. It is a scalar (0D tensor) of type

kFLOAT. It is an optional input with default 0.0f.

460

Class Documentation

The layer has the following outputs, in order of output index:

• SelectedIndices contains the indices of the selected boxes. It is a linear tensor of type kINT32. It has shape

[NumOutputBoxes, 3]. Each row contains a (batchIndex, classIndex, boxIndex) tuple. The output boxes are

sorted in order of increasing batchIndex and then in order of decreasing score within each batchIndex. For each

batchIndex, the ordering of output boxes with the same score is unspecified. If MaxOutputBoxesPerClass is a

constant input, the maximum number of output boxes is batchSize ∗ numClasses ∗ min(numInputBounding←↩

Boxes, MaxOutputBoxesPerClass). Otherwise, the maximum number of output boxes is batchSize ∗ numClasses

∗ numInputBoundingBoxes. The maximum number of output boxes is used to determine the upper-bound on

allocated memory for this output tensor.

• NumOutputBoxes is the number of output boxes in SelectedIndices. It is a scalar (0D tensor) of type kINT32.

Warning

There is a hardware-dependent limit K such that only the K highest scoring boxes in each batch item will be

considered for selection. The value of K is 2000 for SM 5.3 and 6.2 devices, and 5000 otherwise.

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.88.2

Constructor & Destructor Documentation

9.88.2.1

∼INMSLayer()

virtual nvinfer1::INMSLayer::∼INMSLayer ( ) [protected], [virtual], [default], [noexcept]

9.88.3

Member Function Documentation

9.88.3.1

getBoundingBoxFormat()

BoundingBoxFormat nvinfer1::INMSLayer::getBoundingBoxFormat ( ) const [inline], [noexcept]

Get the bounding box format parameter for the layer.

See also

BoundingBoxFormat

setBoundingBoxFormat()

9.88 nvinfer1::INMSLayer Class Reference

461

9.88.3.2

getTopKBoxLimit()

int32 t nvinfer1::INMSLayer::getTopKBoxLimit ( ) const [inline], [noexcept]

Get the TopK box limit parameter for the layer.

See also

setTopKBoxLimit()

9.88.3.3

setBoundingBoxFormat()

void nvinfer1::INMSLayer::setBoundingBoxFormat (

BoundingBoxFormat fmt ) [inline], [noexcept]

Set the bounding box format parameter for the layer.

The default value for the bounding box format parameter is kCORNER PAIRS.

See also

BoundingBoxFormat

getBoundingBoxFormat()

9.88.3.4

setInput()

void nvinfer1::ILayer::setInput (

int32 t index,

ITensor & tensor ) [inline], [noexcept]

Append or replace an input of this layer with a specific tensor.

Parameters

index

the index of the input to modify.

tensor

the new input tensor

The indices are as follows:

•

0: The required Boxes tensor.

462

Class Documentation

• 1: The required Scores tensor.

• 2: The required MaxOutputBoxesPerClass tensor.

• 3: The optional IoUThreshold tensor.

• 4: The optional ScoreThreshold tensor.

If this function is called for an index greater or equal to getNbInputs(), then afterwards getNbInputs() returns index +

1, and any missing intervening inputs are set to null. Note that only optional inputs can be missing.

9.88.3.5

setTopKBoxLimit()

void nvinfer1::INMSLayer::setTopKBoxLimit (

int32 t limit ) [inline], [noexcept]

Set the TopK box limit parameter for the layer.

The TopK box limit is the maximum number of filtered boxes considered for selection per batch item. The default

value for the TopK box limit parameter is 2000 for SM 5.3 and 6.2 devices, and 5000 otherwise. The TopK box limit

must be less than or equal to {2000 for SM 5.3 and 6.2 devices, 5000 otherwise}.

See also

getTopKBoxLimit()

9.88.4

Member Data Documentation

9.88.4.1

mImpl

apiv::VNMSLayer∗ nvinfer1::INMSLayer::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.89 nvinfer1::INoCopy Class Reference

463

9.89

nvinfer1::INoCopy Class Reference

Forward declaration of IEngineInspector for use by other interfaces.

#include <NvInferRuntime.h>

Inheritance diagram for nvinfer1::INoCopy:

nvinfer1::INoCopy

nvinfer1::IAlgorithm

nvinfer1::IAlgorithmContext

nvinfer1::IAlgorithmIOInfo

nvinfer1::IAlgorithmVariant

nvinfer1::IBuilder

nvinfer1::IBuilderConfig

nvinfer1::ICudaEngine

nvinfer1::IDimensionExpr

nvinfer1::IEngineInspector

nvinfer1::IExecutionContext

nvinfer1::IExprBuilder

nvinfer1::IHostMemory

nvinfer1::IIfConditional

nvinfer1::ILayer

nvinfer1::ILoop

nvinfer1::INetworkDefinition

nvinfer1::IOptimizationProfile

nvinfer1::IRefitter

nvinfer1::IRuntime

nvinfer1::ITensor

nvinfer1::ITimingCache

Protected Member Functions

• INoCopy ()=default

• virtual ∼INoCopy ()=default

• INoCopy (INoCopy const &other)=delete

• INoCopy & operator= (INoCopy const &other)=delete

• INoCopy (INoCopy &&other)=delete

• INoCopy & operator= (INoCopy &&other)=delete

9.89.1

Detailed Description

Forward declaration of IEngineInspector for use by other interfaces.

Base class for all TensorRT interfaces that are implemented by the TensorRT libraries

Objects of such classes are not movable or copyable, and should only be manipulated via pointers.

464

Class Documentation

9.89.2

Constructor & Destructor Documentation

9.89.2.1

INoCopy() [1/3]

nvinfer1::INoCopy::INoCopy ( ) [protected], [default]

9.89.2.2

∼INoCopy()

virtual nvinfer1::INoCopy::∼INoCopy ( ) [protected], [virtual], [default]

9.89.2.3

INoCopy() [2/3]

nvinfer1::INoCopy::INoCopy (

INoCopy const & other ) [protected], [delete]

9.89.2.4

INoCopy() [3/3]

nvinfer1::INoCopy::INoCopy (

INoCopy && other ) [protected], [delete]

9.89.3

Member Function Documentation

9.89.3.1

operator=() [1/2]

INoCopy & nvinfer1::INoCopy::operator= (

INoCopy && other ) [protected], [delete]

9.90 INonZero Class Reference

465

9.89.3.2

operator=() [2/2]

INoCopy & nvinfer1::INoCopy::operator= (

INoCopy const & other ) [protected], [delete]

The documentation for this class was generated from the following file:

• NvInferRuntime.h

9.90

INonZero Class Reference

A NonZero layer in a network.

#include <NvInfer.h>

9.90.1

Detailed Description

A NonZero layer in a network.

This layer gets the positions of elements that are non-zero in the input. For boolean input, ”non-zero” means ”true”.

Semantics are similar to ONNX NonZero.

The input may have type kFLOAT, kHALF, kINT32, or kBOOL.

The output is a matrix of type kINT32. For an input with dimensions [L1, L2, ..., Lm], the output has dimensions

[m,n], where n is the number of non-zero elements. I.e., each column denotes a m-D position.

The columns are lexically ordered. E.g., a column with [3,2,4,7] precedes a column with [3,2,5,6].

Tip: ”compress” can be implemented with INonZero+IShuffle+Gather. For example, to compress a tensor x over axis

k using mask vector v, use nonzero(v) to compute the subscripts, shuffle with reshape dimensions = [-1] to make the

subscripts 1D, and then gather with the subscripts.

The documentation for this class was generated from the following file:

• NvInfer.h

9.91

nvinfer1::INonZeroLayer Class Reference

#include <NvInfer.h>

Inheritance diagram for nvinfer1::INonZeroLayer:

nvinfer1::INoCopy

nvinfer1::ILayer

nvinfer1::INonZeroLayer

466

Class Documentation

Protected Member Functions

• virtual ∼INonZeroLayer () noexcept=default

Protected Attributes

• apiv::VNonZeroLayer ∗ mImpl

Additional Inherited Members

9.91.1

Constructor & Destructor Documentation

9.91.1.1

∼INonZeroLayer()

virtual nvinfer1::INonZeroLayer::∼INonZeroLayer ( ) [protected], [virtual], [default], [noexcept]

9.91.2

Member Data Documentation

9.91.2.1

mImpl

apiv::VNonZeroLayer∗ nvinfer1::INonZeroLayer::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.92

nvinfer1::INormalizationLayer Class Reference

A normalization layer in a network definition.

#include <NvInfer.h>

Inheritance diagram for nvinfer1::INormalizationLayer:

nvinfer1::INoCopy

nvinfer1::ILayer

nvinfer1::INormalizationLayer

9.92 nvinfer1::INormalizationLayer Class Reference

467

Public Member Functions

• void setEpsilon (float eps) noexcept

Set the epsilon value used for the normalization calculation.

• float getEpsilon () const noexcept

Get the epsilon value used for the normalization calculation.

• void setAxes (uint32 t axesMask) noexcept

Set the reduction axes for the normalization calculation.

• uint32 t getAxes () const noexcept

Get the axes value used for the normalization calculation.

• void setNbGroups (int32 t nbGroups) noexcept

Set the number of groups used to split the channels in the normalization calculation.

• int32 t getNbGroups () const noexcept

Get the number of groups used to split the channels for the normalization calculation.

• void setComputePrecision (DataType type) noexcept

Set the compute precision of this layer.

• DataType getComputePrecision () const noexcept

Get the compute precision of this layer.

Protected Member Functions

• virtual ∼INormalizationLayer () noexcept=default

Protected Attributes

• apiv::VNormalizationLayer ∗ mImpl

9.92.1

Detailed Description

A normalization layer in a network definition.

The normalization layer performs the following operation:

X - input Tensor Y - output Tensor S - scale Tensor B - bias Tensor

Y = (X - Mean(X, axes)) / Sqrt(Variance(X) + epsilon) ∗ S + B

Where Mean(X, axes) is a reduction over a set of axes, and Variance(X) = Mean((X - Mean(X, axes))^∧ 2, axes).

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.92.2

Constructor & Destructor Documentation

468

Class Documentation

9.92.2.1

∼INormalizationLayer()

virtual nvinfer1::INormalizationLayer::∼INormalizationLayer ( ) [protected], [virtual], [default],

[noexcept]

9.92.3

Member Function Documentation

9.92.3.1

getAxes()

uint32 t nvinfer1::INormalizationLayer::getAxes ( ) const [inline], [noexcept]

Get the axes value used for the normalization calculation.

Returns

The axes used for the normalization calculation.

9.92.3.2

getComputePrecision()

DataType nvinfer1::INormalizationLayer::getComputePrecision ( ) const [inline], [noexcept]

Get the compute precision of this layer.

Returns

The datatype used for the compute precision of this layer.

9.92.3.3

getEpsilon()

float nvinfer1::INormalizationLayer::getEpsilon ( ) const [inline], [noexcept]

Get the epsilon value used for the normalization calculation.

Returns

The epsilon value used for the normalization calculation.

9.92 nvinfer1::INormalizationLayer Class Reference

469

9.92.3.4

getNbGroups()

int32 t nvinfer1::INormalizationLayer::getNbGroups ( ) const [inline], [noexcept]

Get the number of groups used to split the channels for the normalization calculation.

Returns

The number of groups used to split the channel used for the normalization calculation.

9.92.3.5

setAxes()

void nvinfer1::INormalizationLayer::setAxes (

uint32 t axesMask ) [inline], [noexcept]

Set the reduction axes for the normalization calculation.

Parameters

axesMask

The axes used for the normalization calculation.

9.92.3.6

setComputePrecision()

void nvinfer1::INormalizationLayer::setComputePrecision (

DataType type ) [inline], [noexcept]

Set the compute precision of this layer.

Parameters

type

The datatype used for the compute precision of this layer.

By default TensorRT will run the normalization computation in DataType::kFLOAT32 even in mixed precision mode

regardless of any set builder flags to avoid overflow errors. To override this default, use this function to set the desired

compute precision.

setPrecision() and setOutputPrecision() functions can still be called to control the input and output data types to this

layer.

Only DataType::kFLOAT32 and DataType::kHALF are valid types for type.

470

Class Documentation

9.92.3.7

setEpsilon()

void nvinfer1::INormalizationLayer::setEpsilon (

float eps ) [inline], [noexcept]

Set the epsilon value used for the normalization calculation.

The default value of eps is 1e-5F.

Parameters

eps

The epsilon value used for the normalization calculation.

9.92.3.8

setNbGroups()

void nvinfer1::INormalizationLayer::setNbGroups (

int32 t nbGroups ) [inline], [noexcept]

Set the number of groups used to split the channels in the normalization calculation.

The input tensor channels are divided into nbGroups groups, and normalization is performed per group. The channel

dimension is considered to be the second dimension in a [N, C, H, W, ...] formatted tensor.

The default nbGroups is 1.

Warning

It is an error to set nbGroups to a value that does not evenly divide into the number of channels of the

input tensor.

When nbGroups is != 1, it is expected that the provided axesMask will have all bits corresponding to

dimensions after the channel dimension set to 1, with all other bits set to 0.

Parameters

nbGroups

The number of groups to split the channels into for the normalization calculation.

9.92.4

Member Data Documentation

9.92.4.1

mImpl

apiv::VNormalizationLayer∗ nvinfer1::INormalizationLayer::mImpl [protected]

The documentation for this class was generated from the following file:

9.93 nvinfer1::IOneHotLayer Class Reference

471

• NvInfer.h

9.93

nvinfer1::IOneHotLayer Class Reference

A OneHot layer in a network definition.

#include <NvInfer.h>

Inheritance diagram for nvinfer1::IOneHotLayer:

nvinfer1::INoCopy

nvinfer1::ILayer

nvinfer1::IOneHotLayer

Public Member Functions

• void setAxis (int32 t axis) noexcept

Set the axis parameter.

• int32 t getAxis () const noexcept

Get the value of the axis parameter.

Protected Attributes

• apiv::VOneHotLayer ∗ mImpl

Additional Inherited Members

9.93.1

Detailed Description

A OneHot layer in a network definition.

The OneHot layer has three input tensors: Indices, Values, and Depth, one output tensor: Output, and an axis attribute.

• Indices is an Int32 tensor that determines which locations in Output to set as on value.

• Values is a two-element (rank=1) tensor that consists of [off value, on value]

• Depth is an Int32 shape tensor of rank 0, which contains the depth (number of classes) of the one-hot encoding.

The depth tensor must be a build-time constant, and its value should be positive.

472

Class Documentation

• Output is a tensor with rank = rank(indices)+1, where the added dimension contains the one-hot encoding. The

data types of Output is equal to the Values data type.

• Axis is a scaler specifying to which dimension of the output one-hot encoding is added. Axis defaults to -1, that

is the new dimension in the output is its final dimension. Valid range for axis is -rank(indices)-1 <= axis <=

rank(indices).

The output is computed by copying off values to all output elements, then setting on value on the indices specified by

the indices tensor. when axis = 0: output[indices[i, j, k], i, j, k] = on value for all i, j, k and off value otherwise.

when axis = -1: output[i, j, k, indices[i, j, k]] = on value for all i, j, k and off value otherwise.

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.93.2

Member Function Documentation

9.93.2.1

getAxis()

int32 t nvinfer1::IOneHotLayer::getAxis ( ) const [inline], [noexcept]

Get the value of the axis parameter.

9.93.2.2

setAxis()

void nvinfer1::IOneHotLayer::setAxis (

int32 t axis ) [inline], [noexcept]

Set the axis parameter.

See also

IOneHotLayer

9.93.3

Member Data Documentation

9.94 nvonnxparser::IOnnxConfig Class Reference

473

9.93.3.1

mImpl

apiv::VOneHotLayer∗ nvinfer1::IOneHotLayer::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.94

nvonnxparser::IOnnxConfig Class Reference

Configuration Manager Class.

#include <NvOnnxConfig.h>

Public Types

• typedef int32 t Verbosity

Defines Verbosity level.

Public Member Functions

•

virtual ∼IOnnxConfig () noexcept=default

•

virtual void setModelDtype (const nvinfer1::DataType) noexcept=0

Set the Model Data Type.

•

virtual nvinfer1::DataType getModelDtype () const noexcept=0

Get the Model Data Type.

•

virtual char const ∗ getModelFileName () const noexcept=0

Get the Model FileName.

•

virtual void setModelFileName (char const ∗onnxFilename) noexcept=0

Set the Model File Name.

•

virtual Verbosity getVerbosityLevel () const noexcept=0

Get the Verbosity Level.

•

virtual void addVerbosity () noexcept=0

Increase the Verbosity Level.

•

virtual void reduceVerbosity () noexcept=0

Reduce the Verbosity Level.

•

virtual void setVerbosityLevel (Verbosity) noexcept=0

Set to specific verbosity Level.

•

virtual char const ∗ getTextFileName () const noexcept=0

Returns the File Name of the Network Description as a Text File.

•

virtual void setTextFileName (char const ∗textFileName) noexcept=0

Set the File Name of the Network Description as a Text File.

•

virtual char const ∗ getFullTextFileName () const noexcept=0

Get the File Name of the Network Description as a Text File, including the weights.

•

virtual void setFullTextFileName (char const ∗fullTextFileName) noexcept=0

Set the File Name of the Network Description as a Text File, including the weights.

•

virtual bool getPrintLayerInfo () const noexcept=0

Get whether the layer information will be printed.

•

virtual void setPrintLayerInfo (bool) noexcept=0

Set whether the layer information will be printed.

•

virtual TRT DEPRECATED void destroy () noexcept=0

Destroy IOnnxConfig object.

474

Class Documentation

9.94.1

Detailed Description

Configuration Manager Class.

9.94.2

Member Typedef Documentation

9.94.2.1

Verbosity

nvonnxparser::IOnnxConfig::Verbosity

Defines Verbosity level.

9.94.3

Constructor & Destructor Documentation

9.94.3.1

∼IOnnxConfig()

virtual nvonnxparser::IOnnxConfig::∼IOnnxConfig ( ) [virtual], [default], [noexcept]

9.94.4

Member Function Documentation

9.94.4.1

addVerbosity()

virtual void nvonnxparser::IOnnxConfig::addVerbosity ( ) [pure virtual], [noexcept]

Increase the Verbosity Level.

Returns

The Verbosity Level.

See also

reduceVerbosity(), setVerbosity(Verbosity)

9.94.4.2

destroy()

virtual TRT DEPRECATED void nvonnxparser::IOnnxConfig::destroy ( ) [pure virtual], [noexcept]

Destroy IOnnxConfig object.

Deprecated Use delete instead. Deprecated in TRT 8.0.

9.94 nvonnxparser::IOnnxConfig Class Reference

475

Warning

Calling destroy on a managed pointer will result in a double-free error.

9.94.4.3

getFullTextFileName()

virtual char const ∗ nvonnxparser::IOnnxConfig::getFullTextFileName ( ) const [pure virtual],

[noexcept]

Get the File Name of the Network Description as a Text File, including the weights.

Returns

Return the name of the file containing the network description converted to a plain text, used for debugging

purposes.

See also

setFullTextFilename()

9.94.4.4

getModelDtype()

virtual nvinfer1::DataType nvonnxparser::IOnnxConfig::getModelDtype ( ) const [pure virtual],

[noexcept]

Get the Model Data Type.

Returns

the data type of the model.

See also

setModelDtype() and DataType

476

Class Documentation

9.94.4.5

getModelFileName()

virtual char const ∗ nvonnxparser::IOnnxConfig::getModelFileName ( ) const [pure virtual], [noexcept]

Get the Model FileName.

Returns

Return the Model Filename, as a null-terminated C-style string.

See also

setModelFileName()

9.94.4.6

getPrintLayerInfo()

virtual bool nvonnxparser::IOnnxConfig::getPrintLayerInfo ( ) const [pure virtual], [noexcept]

Get whether the layer information will be printed.

Returns

Returns whether the layer information will be printed.

See also

setPrintLayerInfo()

9.94.4.7

getTextFileName()

virtual char const ∗ nvonnxparser::IOnnxConfig::getTextFileName ( ) const [pure virtual], [noexcept]

Returns the File Name of the Network Description as a Text File.

Returns

Return the name of the file containing the network description converted to a plain text, used for debugging

purposes.

See also

setTextFilename()

9.94 nvonnxparser::IOnnxConfig Class Reference

477

9.94.4.8

getVerbosityLevel()

virtual Verbosity nvonnxparser::IOnnxConfig::getVerbosityLevel ( ) const [pure virtual], [noexcept]

Get the Verbosity Level.

Returns

The Verbosity Level.

See also

addVerbosity(), reduceVerbosity()

9.94.4.9

reduceVerbosity()

virtual void nvonnxparser::IOnnxConfig::reduceVerbosity ( ) [pure virtual], [noexcept]

Reduce the Verbosity Level.

See also

addVerbosity(), setVerbosity(Verbosity)

9.94.4.10 setFullTextFileName()

virtual void nvonnxparser::IOnnxConfig::setFullTextFileName (

char const ∗ fullTextFileName ) [pure virtual], [noexcept]

Set the File Name of the Network Description as a Text File, including the weights.

This API allows setting a file name for the network description in plain text, equivalent of the ONNX protobuf.

This method copies the name string.

Parameters

fullTextFileName

Name of the file.

478

Class Documentation

See also

getFullTextFilename()

9.94.4.11 setModelDtype()

virtual void nvonnxparser::IOnnxConfig::setModelDtype (

const nvinfer1::DataType

) [pure virtual], [noexcept]

Set the Model Data Type.

Sets the Model DataType, one of the following: float -d 32 (default), half precision -d 16, and int8 -d 8 data types.

See also

getModelDtype()

9.94.4.12 setModelFileName()

virtual void nvonnxparser::IOnnxConfig::setModelFileName (

char const ∗ onnxFilename ) [pure virtual], [noexcept]

Set the Model File Name.

The Model File name contains the Network Description in ONNX pb format.

This method copies the name string.

Parameters

onnxFilename

The name.

See also

getModelFileName()

9.94.4.13 setPrintLayerInfo()

virtual void nvonnxparser::IOnnxConfig::setPrintLayerInfo (

bool

) [pure virtual], [noexcept]

Set whether the layer information will be printed.

9.94 nvonnxparser::IOnnxConfig Class Reference

479

See also

getPrintLayerInfo()

9.94.4.14 setTextFileName()

virtual void nvonnxparser::IOnnxConfig::setTextFileName (

char const ∗ textFileName ) [pure virtual], [noexcept]

Set the File Name of the Network Description as a Text File.

This API allows setting a file name for the network description in plain text, equivalent of the ONNX protobuf.

This method copies the name string.

Parameters

textFileName

Name of the file.

See also

getTextFilename()

9.94.4.15 setVerbosityLevel()

virtual void nvonnxparser::IOnnxConfig::setVerbosityLevel (

Verbosity

) [pure virtual], [noexcept]

Set to specific verbosity Level.

See also

addVerbosity(), reduceVerbosity()

The documentation for this class was generated from the following file:

• NvOnnxConfig.h

480

Class Documentation

9.95

nvinfer1::IOptimizationProfile Class Reference

Optimization profile for dynamic input dimensions and shape tensors.

#include <NvInferRuntime.h>

Inheritance diagram for nvinfer1::IOptimizationProfile:

nvinfer1::INoCopy

nvinfer1::IOptimizationProfile

Public Member Functions

• bool setDimensions (char const ∗inputName, OptProfileSelector select, Dims dims) noexcept

Set the minimum / optimum / maximum dimensions for a dynamic input tensor.

• Dims getDimensions (char const ∗inputName, OptProfileSelector select) const noexcept

Get the minimum / optimum / maximum dimensions for a dynamic input tensor.

• bool setShapeValues (char const ∗inputName, OptProfileSelector select, int32 t const ∗values, int32 t nbValues)

noexcept

Set the minimum / optimum / maximum values for an input shape tensor.

• int32 t getNbShapeValues (char const ∗inputName) const noexcept

Get the number of values for an input shape tensor.

• int32 t const ∗ getShapeValues (char const ∗inputName, OptProfileSelector select) const noexcept

Get the minimum / optimum / maximum values for an input shape tensor.

• bool setExtraMemoryTarget (float target) noexcept

Set a target for extra GPU memory that may be used by this profile.

• float getExtraMemoryTarget () const noexcept

Get the extra memory target that has been defined for this profile.

• bool isValid () const noexcept

Check whether the optimization profile can be passed to an IBuilderConfig object.

Protected Member Functions

• virtual ∼IOptimizationProfile () noexcept=default

Protected Attributes

• apiv::VOptimizationProfile ∗ mImpl

9.95 nvinfer1::IOptimizationProfile Class Reference

481

9.95.1

Detailed Description

Optimization profile for dynamic input dimensions and shape tensors.

When building an ICudaEngine from an INetworkDefinition that has dynamically resizable inputs (at least one input

tensor has one or more of its dimensions specified as -1) or shape input tensors, users need to specify at least one

optimization profile. Optimization profiles are numbered 0, 1, ... The first optimization profile that has been defined

(with index 0) will be used by the ICudaEngine whenever no optimization profile has been selected explicitly. If

none of the inputs are dynamic, the default optimization profile will be generated automatically unless it is explicitly

provided by the user (this is possible but not required in this case). If more than a single optimization profile is defined,

users may set a target how much additional weight space should be maximally allocated to each additional profile (as

a fraction of the maximum, unconstrained memory).

Users set optimum input tensor dimensions, as well as minimum and maximum input tensor dimensions. The builder

selects the kernels that result in the lowest runtime for the optimum input tensor dimensions, and are valid for all input

tensor sizes in the valid range between minimum and maximum dimensions. A runtime error will be raised if the

input tensor dimensions fall outside the valid range for this profile. Likewise, users provide minimum, optimum, and

maximum values for all shape tensor input values.

See also

IBuilderConfig::addOptimizationProfile()

9.95.2

Constructor & Destructor Documentation

9.95.2.1

∼IOptimizationProfile()

virtual nvinfer1::IOptimizationProfile::∼IOptimizationProfile ( ) [protected], [virtual], [default],

[noexcept]

9.95.3

Member Function Documentation

9.95.3.1

getDimensions()

Dims nvinfer1::IOptimizationProfile::getDimensions (

char const ∗ inputName,

OptProfileSelector select ) const [inline], [noexcept]

Get the minimum / optimum / maximum dimensions for a dynamic input tensor.

If the dimensions have not been previously set via setDimensions(), return an invalid Dims with nbDims == -1.

NVIDIA TensorRT 8.6.11 API Reference for DRIVE OS (May 2023) - page 7