NVIDIA TensorRT 8.6.11 API Reference for DRIVE OS (May 2023)

9.6 nvinfer1::Dims4 Class Reference

107

9.6

nvinfer1::Dims4 Class Reference

Descriptor for four-dimensional data.

#include <NvInferLegacyDims.h>

Inheritance diagram for nvinfer1::Dims4:

nvinfer1::Dims32

nvinfer1::Dims2

nvinfer1::Dims3

nvinfer1::Dims4

Public Member Functions

• Dims4 ()

Construct an empty Dims4 object.

• Dims4 (int32 t d0, int32 t d1, int32 t d2, int32 t d3)

Construct a Dims4 from 4 elements.

Additional Inherited Members

9.6.1

Detailed Description

Descriptor for four-dimensional data.

9.6.2

Constructor & Destructor Documentation

9.6.2.1

Dims4() [1/2]

nvinfer1::Dims4::Dims4 ( ) [inline]

Construct an empty Dims4 object.

9.6.2.2

Dims4() [2/2]

nvinfer1::Dims4::Dims4 (

int32 t d0,

int32 t d1,

int32 t d2,

int32 t d3 ) [inline]

Construct a Dims4 from 4 elements.

108

Class Documentation

Parameters

The first element.

The second element.

The third element.

The fourth element.

The documentation for this class was generated from the following file:

• NvInferLegacyDims.h

9.7

nvinfer1::DimsExprs Class Reference

#include <NvInferRuntime.h>

Public Attributes

• int32 t nbDims

The number of dimensions.

• IDimensionExpr const ∗ d [Dims::MAX DIMS]

The extent of each dimension.

9.7.1

Detailed Description

Analog of class Dims with expressions instead of constants for the dimensions.

9.7.2

Member Data Documentation

9.7.2.1

IDimensionExpr const∗ nvinfer1::DimsExprs::d[Dims::MAX DIMS]

The extent of each dimension.

9.8 nvinfer1::DimsHW Class Reference

109

9.7.2.2

nbDims

int32 t nvinfer1::DimsExprs::nbDims

The number of dimensions.

The documentation for this class was generated from the following file:

• NvInferRuntime.h

9.8

nvinfer1::DimsHW Class Reference

Descriptor for two-dimensional spatial data.

#include <NvInferLegacyDims.h>

Inheritance diagram for nvinfer1::DimsHW:

nvinfer1::Dims32

nvinfer1::Dims2

nvinfer1::DimsHW

Public Member Functions

• DimsHW ()

Construct an empty DimsHW object.

• DimsHW (int32 t height, int32 t width)

Construct a DimsHW given height and width.

• int32 t & h ()

Get the height.

• int32 t h () const

Get the height.

• int32 t & w ()

Get the width.

• int32 t w () const

Get the width.

Additional Inherited Members

9.8.1

Detailed Description

Descriptor for two-dimensional spatial data.

112

Class Documentation

Public Attributes

• PluginTensorDesc desc

Information required to interpret a pointer to tensor data, except that desc.dims has -1 in place of any runtime dimension.

• Dims min

Lower bounds on tensor’s dimensions.

• Dims max

Upper bounds on tensor’s dimensions.

9.9.1

Detailed Description

Summarizes tensors that a plugin might see for an input or output.

9.9.2

Member Data Documentation

9.9.2.1

desc

PluginTensorDesc nvinfer1::DynamicPluginTensorDesc::desc

Information required to interpret a pointer to tensor data, except that desc.dims has -1 in place of any runtime dimen-

sion.

9.9.2.2

max

Dims nvinfer1::DynamicPluginTensorDesc::max

Upper bounds on tensor’s dimensions.

9.9.2.3

min

Dims nvinfer1::DynamicPluginTensorDesc::min

Lower bounds on tensor’s dimensions.

The documentation for this class was generated from the following file:

• NvInferRuntime.h

9.10 nvinfer1::impl::EnumMaxImpl< T > Struct Template Reference

113

9.10

nvinfer1::impl::EnumMaxImpl< T > Struct Template Reference

Declaration of EnumMaxImpl struct to store maximum number of elements in an enumeration type.

9.10.1

Detailed Description

template<typename T>

struct nvinfer1::impl::EnumMaxImpl< T >

Declaration of EnumMaxImpl struct to store maximum number of elements in an enumeration type.

The documentation for this struct was generated from the following file:

• NvInferRuntimeBase.h

9.11

nvinfer1::impl::EnumMaxImpl< ActivationType > Struct Reference

#include <NvInfer.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 12

9.11.1

Detailed Description

Maximum number of elements in ActivationType enum.

See also

ActivationType

9.11.2

Member Data Documentation

9.11.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< ActivationType >::kVALUE = 12 [static], [constexpr]

The documentation for this struct was generated from the following file:

• NvInfer.h

114

Class Documentation

9.12

nvinfer1::impl::EnumMaxImpl< AllocatorFlag > Struct Reference

Maximum number of elements in AllocatorFlag enum.

#include <NvInferRuntimeBase.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 1

maximum number of elements in AllocatorFlag enum

9.12.1

Detailed Description

Maximum number of elements in AllocatorFlag enum.

See also

AllocatorFlag

9.12.2

Member Data Documentation

9.12.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< AllocatorFlag >::kVALUE = 1 [static], [constexpr]

maximum number of elements in AllocatorFlag enum

The documentation for this struct was generated from the following file:

• NvInferRuntimeBase.h

9.13

nvinfer1::impl::EnumMaxImpl< DataType > Struct Reference

Maximum number of elements in DataType enum.

#include <NvInferRuntimeBase.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 7

9.14 nvinfer1::impl::EnumMaxImpl< ElementWiseOperation > Struct Reference

115

9.13.1

Detailed Description

Maximum number of elements in DataType enum.

See also

DataType

9.13.2

Member Data Documentation

9.13.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< DataType >::kVALUE = 7 [static], [constexpr]

The documentation for this struct was generated from the following file:

• NvInferRuntimeBase.h

9.14

nvinfer1::impl::EnumMaxImpl< ElementWiseOperation > Struct

Reference

#include <NvInfer.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 14

9.14.1

Detailed Description

Maximum number of elements in ElementWiseOperation enum.

See also

ElementWiseOperation

9.14.2

Member Data Documentation

116

Class Documentation

9.14.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< ElementWiseOperation >::kVALUE = 14 [static],

[constexpr]

The documentation for this struct was generated from the following file:

• NvInfer.h

9.15

nvinfer1::impl::EnumMaxImpl< EngineCapability > Struct

Reference

Maximum number of elements in EngineCapability enum.

#include <NvInferRuntime.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 3

9.15.1

Detailed Description

Maximum number of elements in EngineCapability enum.

See also

EngineCapability

9.15.2

Member Data Documentation

9.15.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< EngineCapability >::kVALUE = 3 [static], [constexpr]

The documentation for this struct was generated from the following file:

• NvInferRuntime.h

9.16 nvinfer1::impl::EnumMaxImpl< ErrorCode > Struct Reference

117

9.16

nvinfer1::impl::EnumMaxImpl< ErrorCode > Struct Reference

Maximum number of elements in ErrorCode enum.

#include <NvInferRuntimeBase.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 11

Declaration of kVALUE.

9.16.1

Detailed Description

Maximum number of elements in ErrorCode enum.

See also

ErrorCode

9.16.2

Member Data Documentation

9.16.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< ErrorCode >::kVALUE = 11 [static], [constexpr]

Declaration of kVALUE.

The documentation for this struct was generated from the following file:

• NvInferRuntimeBase.h

9.17

nvinfer1::impl::EnumMaxImpl< HardwareCompatibilityLevel >

Struct Reference

#include <NvInfer.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 2

118

Class Documentation

9.17.1

Detailed Description

Maximum number of elements in HardwareCompatibilityLevel enum.

See also

HardwareCompatibilityLevel

9.17.2

Member Data Documentation

9.17.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< HardwareCompatibilityLevel >::kVALUE = 2 [static],

[constexpr]

The documentation for this struct was generated from the following file:

• NvInfer.h

9.18

nvinfer1::impl::EnumMaxImpl< ILogger::Severity > Struct

Reference

Maximum number of elements in ILogger::Severity enum.

#include <NvInferRuntimeBase.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 5

Declaration of kVALUE that represents maximum number of elements in ILogger::Severity enum.

9.18.1

Detailed Description

Maximum number of elements in ILogger::Severity enum.

See also

ILogger::Severity

9.19 nvinfer1::impl::EnumMaxImpl< InterpolationMode > Struct Reference

119

9.18.2

Member Data Documentation

9.18.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< ILogger::Severity >::kVALUE = 5 [static], [constexpr]

Declaration of kVALUE that represents maximum number of elements in ILogger::Severity enum.

The documentation for this struct was generated from the following file:

• NvInferRuntimeBase.h

9.19

nvinfer1::impl::EnumMaxImpl< InterpolationMode > Struct

Reference

#include <NvInfer.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 3

9.19.1

Detailed Description

Maximum number of elements in InterpolationMode enum.

See also

InterpolationMode

9.19.2

Member Data Documentation

9.19.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< InterpolationMode >::kVALUE = 3 [static], [constexpr]

The documentation for this struct was generated from the following file:

• NvInfer.h

120

Class Documentation

9.20

nvinfer1::impl::EnumMaxImpl< PaddingMode > Struct Reference

#include <NvInfer.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 6

9.20.1

Detailed Description

Maximum number of elements in PaddingMode enum.

See also

PaddingMode

9.20.2

Member Data Documentation

9.20.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< PaddingMode >::kVALUE = 6 [static], [constexpr]

The documentation for this struct was generated from the following file:

• NvInfer.h

9.21

nvinfer1::impl::EnumMaxImpl< PoolingType > Struct Reference

#include <NvInfer.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 3

9.21.1

Detailed Description

Maximum number of elements in PoolingType enum.

See also

PoolingType

9.22 nvinfer1::impl::EnumMaxImpl< PreviewFeature > Struct Reference

121

9.21.2

Member Data Documentation

9.21.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< PoolingType >::kVALUE = 3 [static], [constexpr]

The documentation for this struct was generated from the following file:

• NvInfer.h

9.22

nvinfer1::impl::EnumMaxImpl< PreviewFeature > Struct Reference

#include <NvInfer.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 3

9.22.1

Detailed Description

Maximum number of elements in PreviewFeature enum.

See also

PreviewFeature

9.22.2

Member Data Documentation

9.22.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< PreviewFeature >::kVALUE = 3 [static], [constexpr]

The documentation for this struct was generated from the following file:

• NvInfer.h

122

Class Documentation

9.23

nvinfer1::impl::EnumMaxImpl< ResizeCoordinateTransformation >

Struct Reference

#include <NvInfer.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 3

9.23.1

Detailed Description

Maximum number of elements in ResizeCoordinateTransformation enum.

See also

ResizeCoordinateTransformation

9.23.2

Member Data Documentation

9.23.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< ResizeCoordinateTransformation >::kVALUE = 3 [static],

[constexpr]

The documentation for this struct was generated from the following file:

• NvInfer.h

9.24

nvinfer1::impl::EnumMaxImpl< ResizeRoundMode > Struct

Reference

#include <NvInfer.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 4

9.25 nvinfer1::impl::EnumMaxImpl< ResizeSelector > Struct Reference

123

9.24.1

Detailed Description

Maximum number of elements in ResizeRoundMode enum.

See also

ResizeRoundMode

9.24.2

Member Data Documentation

9.24.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< ResizeRoundMode >::kVALUE = 4 [static], [constexpr]

The documentation for this struct was generated from the following file:

• NvInfer.h

9.25

nvinfer1::impl::EnumMaxImpl< ResizeSelector > Struct Reference

#include <NvInfer.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 2

9.25.1

Detailed Description

Maximum number of elements in ResizeSelector enum.

See also

ResizeSelector

9.25.2

Member Data Documentation

124

Class Documentation

9.25.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< ResizeSelector >::kVALUE = 2 [static], [constexpr]

The documentation for this struct was generated from the following file:

• NvInfer.h

9.26

nvinfer1::impl::EnumMaxImpl< TensorFormat > Struct Reference

Maximum number of elements in TensorFormat enum.

#include <NvInferRuntimeBase.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 13

Declaration of kVALUE that represents maximum number of elements in TensorFormat enum.

9.26.1

Detailed Description

Maximum number of elements in TensorFormat enum.

See also

TensorFormat

9.26.2

Member Data Documentation

9.26.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< TensorFormat >::kVALUE = 13 [static], [constexpr]

Declaration of kVALUE that represents maximum number of elements in TensorFormat enum.

The documentation for this struct was generated from the following file:

• NvInferRuntimeBase.h

9.27 nvinfer1::impl::EnumMaxImpl< TensorIOMode > Struct Reference

125

9.27

nvinfer1::impl::EnumMaxImpl< TensorIOMode > Struct Reference

Maximum number of elements in TensorIOMode enum.

#include <NvInferRuntimeBase.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 3

9.27.1

Detailed Description

Maximum number of elements in TensorIOMode enum.

See also

TensorIOMode

9.27.2

Member Data Documentation

9.27.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< TensorIOMode >::kVALUE = 3 [static], [constexpr]

The documentation for this struct was generated from the following file:

• NvInferRuntimeBase.h

9.28

nvinfer1::impl::EnumMaxImpl< TensorLocation > Struct Reference

Maximum number of elements in TensorLocation enum.

#include <NvInferRuntime.h>

Static Public Attributes

• static constexpr int32 t kVALUE = 2

126

Class Documentation

9.28.1

Detailed Description

Maximum number of elements in TensorLocation enum.

See also

TensorLocation

9.28.2

Member Data Documentation

9.28.2.1

kVALUE

constexpr int32 t nvinfer1::impl::EnumMaxImpl< TensorLocation >::kVALUE = 2 [static], [constexpr]

The documentation for this struct was generated from the following file:

• NvInferRuntime.h

9.29

nvuffparser::FieldCollection Struct Reference

#include <NvUffParser.h>

Public Attributes

• int32 t nbFields

• FieldMap const ∗ fields

9.29.1

Member Data Documentation

9.29.1.1

fields

FieldMap const∗ nvuffparser::FieldCollection::fields

9.30 nvuffparser::FieldMap Class Reference

127

9.29.1.2

nbFields

int32 t nvuffparser::FieldCollection::nbFields

The documentation for this struct was generated from the following file:

• NvUffParser.h

9.30

nvuffparser::FieldMap Class Reference

An array of field params used as a layer parameter for plugin layers.

#include <NvUffParser.h>

Public Member Functions

• TRT DEPRECATED FieldMap (char const ∗name, void const ∗data, FieldType const type, int32 t length=1)

• FieldMap ()=default

Default constructor.

Public Attributes

• char const ∗ name {}

• void const ∗ data {}

• FieldType type {FieldType::kUNKNOWN}

• int32 t length {1}

9.30.1

Detailed Description

An array of field params used as a layer parameter for plugin layers.

The node fields are passed by the parser to the API through the plugin constructor. The implementation of the plugin

should parse the contents of the fieldMap as part of the plugin constructor

9.30.2

Constructor & Destructor Documentation

128

Class Documentation

9.30.2.1

FieldMap() [1/2]

TRT DEPRECATED nvuffparser::FieldMap::FieldMap (

char const ∗ name,

void const ∗ data,

FieldType const type,

int32 t length = 1 )

Deprecated Legacy constructor, retained for ABI compatibility. Deprecated in TensorRT 8.6. Use the default con-

structor instead.

9.30.2.2

FieldMap() [2/2]

nvuffparser::FieldMap::FieldMap ( ) [default]

Default constructor.

9.30.3

Member Data Documentation

9.30.3.1

data

void const∗ nvuffparser::FieldMap::data {}

9.30.3.2

length

int32 t nvuffparser::FieldMap::length {1}

9.30.3.3

name

char const∗ nvuffparser::FieldMap::name {}

9.31 nvinfer1::plugin::GridAnchorParameters Struct Reference

129

9.30.3.4

type

FieldType nvuffparser::FieldMap::type {FieldType::kUNKNOWN}

The documentation for this class was generated from the following file:

• NvUffParser.h

9.31

nvinfer1::plugin::GridAnchorParameters Struct Reference

The Anchor Generator plugin layer generates the prior boxes of designated sizes and aspect ratios across all dimensions

(H x W). GridAnchorParameters defines a set of parameters for creating the plugin layer for all feature maps. It

contains:

#include <NvInferPluginUtils.h>

Public Attributes

• float minSize

• float maxSize

• float ∗ aspectRatios

• int32 t numAspectRatios

• int32 t H

• int32 t W

• float variance [4]

9.31.1

Detailed Description

The Anchor Generator plugin layer generates the prior boxes of designated sizes and aspect ratios across all dimensions

(H x W). GridAnchorParameters defines a set of parameters for creating the plugin layer for all feature maps. It

contains:

Parameters

minScale

Scale of anchors corresponding to finest resolution.

maxScale

Scale of anchors corresponding to coarsest resolution.

aspectRatios

List of aspect ratios to place on each grid point.

numAspectRatios

Number of elements in aspectRatios.

Height of feature map to generate anchors for.

Width of feature map to generate anchors for.

variance

Variance for adjusting the prior boxes.

9.32 nvinfer1::IActivationLayer Class Reference

131

9.31.2.7

int32 t nvinfer1::plugin::GridAnchorParameters::W

The documentation for this struct was generated from the following file:

• NvInferPluginUtils.h

9.32

nvinfer1::IActivationLayer Class Reference

An Activation layer in a network definition.

#include <NvInfer.h>

Inheritance diagram for nvinfer1::IActivationLayer:

nvinfer1::INoCopy

nvinfer1::ILayer

nvinfer1::IActivationLayer

Public Member Functions

• void setActivationType (ActivationType type) noexcept

Set the type of activation to be performed.

• ActivationType getActivationType () const noexcept

Get the type of activation to be performed.

• void setAlpha (float alpha) noexcept

Set the alpha parameter (must be finite).

• void setBeta (float beta) noexcept

Set the beta parameter (must be finite).

• float getAlpha () const noexcept

Get the alpha parameter.

• float getBeta () const noexcept

Get the beta parameter.

Protected Member Functions

• virtual ∼IActivationLayer () noexcept=default

132

Class Documentation

Protected Attributes

• apiv::VActivationLayer ∗ mImpl

9.32.1

Detailed Description

An Activation layer in a network definition.

This layer applies a per-element activation function to its input.

The output has the same shape as the input.

The input is a shape tensor if the output is a shape tensor.

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.32.2

Constructor & Destructor Documentation

9.32.2.1

∼IActivationLayer()

virtual nvinfer1::IActivationLayer::∼IActivationLayer ( ) [protected], [virtual], [default],

[noexcept]

9.32.3

Member Function Documentation

9.32.3.1

getActivationType()

ActivationType nvinfer1::IActivationLayer::getActivationType ( ) const [inline], [noexcept]

Get the type of activation to be performed.

See also

setActivationType(), ActivationType

9.32 nvinfer1::IActivationLayer Class Reference

133

9.32.3.2

getAlpha()

float nvinfer1::IActivationLayer::getAlpha ( ) const [inline], [noexcept]

Get the alpha parameter.

See also

getBeta(), setAlpha()

9.32.3.3

getBeta()

float nvinfer1::IActivationLayer::getBeta ( ) const [inline], [noexcept]

Get the beta parameter.

See also

getAlpha(), setBeta()

9.32.3.4

setActivationType()

void nvinfer1::IActivationLayer::setActivationType (

ActivationType type ) [inline], [noexcept]

Set the type of activation to be performed.

On the DLA, the valid activation types are kRELU, kSIGMOID, kTANH, and kCLIP.

See also

getActivationType(), ActivationType

134

Class Documentation

9.32.3.5

setAlpha()

void nvinfer1::IActivationLayer::setAlpha (

float alpha ) [inline], [noexcept]

Set the alpha parameter (must be finite).

This parameter is used by the following activations: LeakyRelu, Elu, Selu, Softplus, Clip, HardSigmoid, ScaledTanh,

ThresholdedRelu.

It is ignored by the other activations.

See also

getAlpha(), setBeta()

9.32.3.6

setBeta()

void nvinfer1::IActivationLayer::setBeta (

float beta ) [inline], [noexcept]

Set the beta parameter (must be finite).

This parameter is used by the following activations: Selu, Softplus, Clip, HardSigmoid, ScaledTanh.

It is ignored by the other activations.

See also

getBeta(), setAlpha()

9.32.4

Member Data Documentation

9.32.4.1

mImpl

apiv::VActivationLayer∗ nvinfer1::IActivationLayer::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.33 nvinfer1::IAlgorithm Class Reference

135

9.33

nvinfer1::IAlgorithm Class Reference

Describes a variation of execution of a layer.

An algorithm is represented by IAlgorithmVariant and the

IAlgorithmIOInfo for each of its inputs and outputs. An algorithm can be selected or reproduced using Algorithm←↩

Selector::selectAlgorithms().”.

#include <NvInfer.h>

Inheritance diagram for nvinfer1::IAlgorithm:

nvinfer1::INoCopy

nvinfer1::IAlgorithm

Public Member Functions

• TRT DEPRECATED IAlgorithmIOInfo const & getAlgorithmIOInfo (int32 t index) const noexcept

Returns the format of an Algorithm input or output. Algorithm inputs are incrementally numbered first, followed by

algorithm outputs.

• IAlgorithmVariant const & getAlgorithmVariant () const noexcept

Returns the algorithm variant.

• float getTimingMSec () const noexcept

The time in milliseconds to execute the algorithm.

• std::size t getWorkspaceSize () const noexcept

The size of the GPU temporary memory in bytes which the algorithm uses at execution time.

• IAlgorithmIOInfo const ∗ getAlgorithmIOInfoByIndex (int32 t index) const noexcept

Returns the format of an Algorithm input or output. Algorithm inputs are incrementally numbered first, followed by

algorithm outputs.

Protected Member Functions

• virtual ∼IAlgorithm () noexcept=default

Protected Attributes

• apiv::VAlgorithm ∗ mImpl

9.33.1

Detailed Description

Describes a variation of execution of a layer.

An algorithm is represented by IAlgorithmVariant and the

IAlgorithmIOInfo for each of its inputs and outputs. An algorithm can be selected or reproduced using Algorithm←↩

Selector::selectAlgorithms().”.

See also

IAlgorithmIOInfo, IAlgorithmVariant, IAlgorithmSelector::selectAlgorithms()

136

Class Documentation

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.33.2

Constructor & Destructor Documentation

9.33.2.1

∼IAlgorithm()

virtual nvinfer1::IAlgorithm::∼IAlgorithm ( ) [protected], [virtual], [default], [noexcept]

9.33.3

Member Function Documentation

9.33.3.1

getAlgorithmIOInfo()

TRT DEPRECATED IAlgorithmIOInfo const & nvinfer1::IAlgorithm::getAlgorithmIOInfo (

int32 t index ) const [inline], [noexcept]

Returns the format of an Algorithm input or output. Algorithm inputs are incrementally numbered first, followed by

algorithm outputs.

Parameters

index

Index of the input or output of the algorithm. Incremental numbers assigned to indices of inputs and the

outputs.

Returns

a reference to IAlgorithmIOInfo specified by index or the first algorithm if index is out of range.

Deprecated Deprecated in TensorRT 8.0. Superseded by IAlgorithm::getAlgorithmIOInfoByIndex().

9.33.3.2

getAlgorithmIOInfoByIndex()

IAlgorithmIOInfo const ∗ nvinfer1::IAlgorithm::getAlgorithmIOInfoByIndex (

int32 t index ) const [inline], [noexcept]

Returns the format of an Algorithm input or output. Algorithm inputs are incrementally numbered first, followed by

algorithm outputs.

9.33 nvinfer1::IAlgorithm Class Reference

137

Parameters

index

Index of the input or output of the algorithm. Incremental numbers assigned to indices of inputs and the

outputs.

Returns

a pointer to a IAlgorithmIOInfo interface or nullptr if index is out of range.

9.33.3.3

getAlgorithmVariant()

IAlgorithmVariant const & nvinfer1::IAlgorithm::getAlgorithmVariant ( ) const [inline], [noexcept]

Returns the algorithm variant.

9.33.3.4

getTimingMSec()

float nvinfer1::IAlgorithm::getTimingMSec ( ) const [inline], [noexcept]

The time in milliseconds to execute the algorithm.

9.33.3.5

getWorkspaceSize()

std::size t nvinfer1::IAlgorithm::getWorkspaceSize ( ) const [inline], [noexcept]

The size of the GPU temporary memory in bytes which the algorithm uses at execution time.

9.33.4

Member Data Documentation

9.33.4.1

mImpl

apiv::VAlgorithm∗ nvinfer1::IAlgorithm::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

138

Class Documentation

9.34

nvinfer1::IAlgorithmContext Class Reference

Describes the context and requirements, that could be fulfilled by one or more instances of IAlgorithm.

#include <NvInfer.h>

Inheritance diagram for nvinfer1::IAlgorithmContext:

nvinfer1::INoCopy

nvinfer1::IAlgorithmContext

Public Member Functions

• char const ∗ getName () const noexcept

Return name of the algorithm node. This is a unique identifier for the IAlgorithmContext.

• Dims getDimensions (int32 t index, OptProfileSelector select) const noexcept

Get the minimum / optimum / maximum dimensions for input or output tensor.

• int32 t getNbInputs () const noexcept

Return number of inputs of the algorithm.

• int32 t getNbOutputs () const noexcept

Return number of outputs of the algorithm.

Protected Member Functions

• virtual ∼IAlgorithmContext () noexcept=default

Protected Attributes

• apiv::VAlgorithmContext ∗ mImpl

9.34.1

Detailed Description

Describes the context and requirements, that could be fulfilled by one or more instances of IAlgorithm.

See also

IAlgorithm

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.34 nvinfer1::IAlgorithmContext Class Reference

139

9.34.2

Constructor & Destructor Documentation

9.34.2.1

∼IAlgorithmContext()

virtual nvinfer1::IAlgorithmContext::∼IAlgorithmContext ( ) [protected], [virtual], [default],

[noexcept]

9.34.3

Member Function Documentation

9.34.3.1

getDimensions()

Dims nvinfer1::IAlgorithmContext::getDimensions (

int32 t index,

OptProfileSelector select ) const [inline], [noexcept]

Get the minimum / optimum / maximum dimensions for input or output tensor.

Parameters

index

Index of the input or output of the algorithm. Incremental numbers assigned to indices of inputs and the

outputs.

select

Which of the minimum, optimum, or maximum dimensions to be queried.

9.34.3.2

getName()

char const ∗ nvinfer1::IAlgorithmContext::getName ( ) const [inline], [noexcept]

Return name of the algorithm node. This is a unique identifier for the IAlgorithmContext.

9.34.3.3

getNbInputs()

int32 t nvinfer1::IAlgorithmContext::getNbInputs ( ) const [inline], [noexcept]

Return number of inputs of the algorithm.

140

Class Documentation

9.34.3.4

getNbOutputs()

int32 t nvinfer1::IAlgorithmContext::getNbOutputs ( ) const [inline], [noexcept]

Return number of outputs of the algorithm.

9.34.4

Member Data Documentation

9.34.4.1

mImpl

apiv::VAlgorithmContext∗ nvinfer1::IAlgorithmContext::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.35

nvinfer1::IAlgorithmIOInfo Class Reference

Carries information about input or output of the algorithm. IAlgorithmIOInfo for all the input and output along

with IAlgorithmVariant denotes the variation of algorithm and can be used to select or reproduce an algorithm us-

ing IAlgorithmSelector::selectAlgorithms().

#include <NvInfer.h>

Inheritance diagram for nvinfer1::IAlgorithmIOInfo:

nvinfer1::INoCopy

nvinfer1::IAlgorithmIOInfo

Public Member Functions

• TRT DEPRECATED TensorFormat getTensorFormat () const noexcept

Return TensorFormat of the input/output of algorithm.

• DataType getDataType () const noexcept

Return DataType of the input/output of algorithm.

• Dims getStrides () const noexcept

Return strides of the input/output tensor of algorithm. For vectorized formats, strides are given in units of vectors.

• int64 t getVectorizedDim () const noexcept

Return the index of the vectorized dimension or -1 for non-vectorized formats.

• int64 t getComponentsPerElement () const noexcept

Return the number of components per element. This is always 1 for non-vectorized formats.

9.35 nvinfer1::IAlgorithmIOInfo Class Reference

141

Protected Member Functions

• virtual ∼IAlgorithmIOInfo () noexcept=default

Protected Attributes

• apiv::VAlgorithmIOInfo ∗ mImpl

9.35.1

Detailed Description

Carries information about input or output of the algorithm. IAlgorithmIOInfo for all the input and output along

with IAlgorithmVariant denotes the variation of algorithm and can be used to select or reproduce an algorithm us-

ing IAlgorithmSelector::selectAlgorithms().

See also

IAlgorithmVariant, IAlgorithm, IAlgorithmSelector::selectAlgorithms()

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.35.2

Constructor & Destructor Documentation

9.35.2.1

∼IAlgorithmIOInfo()

virtual nvinfer1::IAlgorithmIOInfo::∼IAlgorithmIOInfo ( ) [protected], [virtual], [default],

[noexcept]

9.35.3

Member Function Documentation

9.35.3.1

getComponentsPerElement()

int64 t nvinfer1::IAlgorithmIOInfo::getComponentsPerElement ( ) const [inline], [noexcept]

Return the number of components per element. This is always 1 for non-vectorized formats.

Returns

the number of components per element.

142

Class Documentation

9.35.3.2

getDataType()

DataType nvinfer1::IAlgorithmIOInfo::getDataType ( ) const [inline], [noexcept]

Return DataType of the input/output of algorithm.

Returns

the data type.

9.35.3.3

getStrides()

Dims nvinfer1::IAlgorithmIOInfo::getStrides ( ) const [inline], [noexcept]

Return strides of the input/output tensor of algorithm. For vectorized formats, strides are given in units of vectors.

Returns

the strides of the tensor.

9.35.3.4

getTensorFormat()

TRT DEPRECATED TensorFormat nvinfer1::IAlgorithmIOInfo::getTensorFormat ( ) const [inline], [noexcept]

Return TensorFormat of the input/output of algorithm.

Deprecated Deprecated in TensorRT 8.6. The strides, data type, and vectorization information is sufficient to uniquely

identify tensor formats.

Returns

the tensor format

9.35.3.5

getVectorizedDim()

int64 t nvinfer1::IAlgorithmIOInfo::getVectorizedDim ( ) const [inline], [noexcept]

Return the index of the vectorized dimension or -1 for non-vectorized formats.

Returns

the index of the vectorized dimension.

9.36 nvinfer1::IAlgorithmSelector Class Reference

143

9.35.4

Member Data Documentation

9.35.4.1

mImpl

apiv::VAlgorithmIOInfo∗ nvinfer1::IAlgorithmIOInfo::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.36

nvinfer1::IAlgorithmSelector Class Reference

Interface implemented by application for selecting and reporting algorithms of a layer provided by the builder.

#include <NvInfer.h>

Public Member Functions

• virtual int32 t selectAlgorithms (IAlgorithmContext const &context, IAlgorithm const ∗const ∗choices, int32 t

nbChoices, int32 t ∗selection) noexcept=0

Select Algorithms for a layer from the given list of algorithm choices.

• virtual void reportAlgorithms (IAlgorithmContext const ∗const ∗algoContexts, IAlgorithm const ∗const ∗algo←↩

Choices, int32 t nbAlgorithms) noexcept=0

Called by TensorRT to report choices it made.

• virtual ∼IAlgorithmSelector () noexcept=default

9.36.1

Detailed Description

Interface implemented by application for selecting and reporting algorithms of a layer provided by the builder.

Note

A layer in context of algorithm selection may be different from ILayer in INetworkDefiniton. For example, an

algorithm might be implementing a conglomeration of multiple ILayers in INetworkDefinition.

9.36.2

Constructor & Destructor Documentation

144

Class Documentation

9.36.2.1

∼IAlgorithmSelector()

virtual nvinfer1::IAlgorithmSelector::∼IAlgorithmSelector ( ) [virtual], [default], [noexcept]

9.36.3

Member Function Documentation

9.36.3.1

reportAlgorithms()

virtual void nvinfer1::IAlgorithmSelector::reportAlgorithms (

IAlgorithmContext const ∗const ∗ algoContexts,

IAlgorithm const ∗const ∗ algoChoices,

int32 t nbAlgorithms ) [pure virtual], [noexcept]

Called by TensorRT to report choices it made.

Note

For a given optimization profile, this call comes after all calls to selectAlgorithms. algoChoices[i] is the choice

that TensorRT made for algoContexts[i], for i in [0, nbAlgorithms-1]

Parameters

algoContexts

The list of all algorithm contexts.

algoChoices

The list of algorithm choices made by TensorRT

nbAlgorithms

The size of algoContexts as well as algoChoices.

9.36.3.2

selectAlgorithms()

virtual int32 t nvinfer1::IAlgorithmSelector::selectAlgorithms (

IAlgorithmContext const & context,

IAlgorithm const ∗const ∗ choices,

int32 t nbChoices,

int32 t ∗ selection ) [pure virtual], [noexcept]

Select Algorithms for a layer from the given list of algorithm choices.

Returns

The number of choices selected from [0, nbChoices-1].

9.37 nvinfer1::IAlgorithmVariant Class Reference

145

Parameters

context

The context for which the algorithm choices are valid.

choices

The list of algorithm choices to select for implementation of this layer.

nbChoices

Number of algorithm choices.

selection

The user writes indices of selected choices in to selection buffer which is of size nbChoices.

Note

TensorRT uses its default algorithm selection to choose from the list provided. If return value is 0, TensorRT's

default algorithm selection is used unless BuilderFlag::kREJECT EMPTY ALGORITHMS (or the deprecated

BuilderFlag::kSTRICT TYPES) is set. The list of choices is valid only for this specific algorithm context.

The documentation for this class was generated from the following file:

• NvInfer.h

9.37

nvinfer1::IAlgorithmVariant Class Reference

provides a unique 128-bit identifier, which along with the input and output information denotes the variation of algo-

rithm and can be used to select or reproduce an algorithm, using IAlgorithmSelector::selectAlgorithms()

#include <NvInfer.h>

Inheritance diagram for nvinfer1::IAlgorithmVariant:

nvinfer1::INoCopy

nvinfer1::IAlgorithmVariant

Public Member Functions

• int64 t getImplementation () const noexcept

Return implementation of the algorithm.

• int64 t getTactic () const noexcept

Return tactic of the algorithm.

Protected Member Functions

• virtual ∼IAlgorithmVariant () noexcept=default

146

Class Documentation

Protected Attributes

• apiv::VAlgorithmVariant ∗ mImpl

9.37.1

Detailed Description

provides a unique 128-bit identifier, which along with the input and output information denotes the variation of algo-

rithm and can be used to select or reproduce an algorithm, using IAlgorithmSelector::selectAlgorithms()

See also

IAlgorithmIOInfo, IAlgorithm, IAlgorithmSelector::selectAlgorithms()

Note

A single implementation can have multiple tactics.

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.37.2

Constructor & Destructor Documentation

9.37.2.1

∼IAlgorithmVariant()

virtual nvinfer1::IAlgorithmVariant::∼IAlgorithmVariant ( ) [protected], [virtual], [default],

[noexcept]

9.37.3

Member Function Documentation

9.37.3.1

getImplementation()

int64 t nvinfer1::IAlgorithmVariant::getImplementation ( ) const [inline], [noexcept]

Return implementation of the algorithm.

9.38 nvinfer1::IAssertionLayer Class Reference

147

9.37.3.2

getTactic()

int64 t nvinfer1::IAlgorithmVariant::getTactic ( ) const [inline], [noexcept]

Return tactic of the algorithm.

9.37.4

Member Data Documentation

9.37.4.1

mImpl

apiv::VAlgorithmVariant∗ nvinfer1::IAlgorithmVariant::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.38

nvinfer1::IAssertionLayer Class Reference

An assertion layer in a network.

#include <NvInfer.h>

Inheritance diagram for nvinfer1::IAssertionLayer:

nvinfer1::INoCopy

nvinfer1::ILayer

nvinfer1::IAssertionLayer

Public Member Functions

• void setMessage (char const ∗message) noexcept

Set the message to print if the assertion fails.

• char const ∗ getMessage () const noexcept

Return the assertion message.

Protected Member Functions

• virtual ∼IAssertionLayer () noexcept=default

148

Class Documentation

Protected Attributes

• apiv::VAssertionLayer ∗ mImpl

9.38.1

Detailed Description

An assertion layer in a network.

The layer has a single input and no output. The input must be a boolean shape tensor. If any element of the input is

provably false at build time, the network is rejected. If any element of the input is false at runtime for the supplied

runtime dimensions, an error occurs, much the same as if any other runtime error (e.g. using IShuffleLayer to change

the volume of a tensor) is handled.

Asserting equality of input dimensions may help the optimizer.

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.38.2

Constructor & Destructor Documentation

9.38.2.1

∼IAssertionLayer()

virtual nvinfer1::IAssertionLayer::∼IAssertionLayer ( ) [protected], [virtual], [default], [noexcept]

9.38.3

Member Function Documentation

9.38.3.1

getMessage()

char const ∗ nvinfer1::IAssertionLayer::getMessage ( ) const [inline], [noexcept]

Return the assertion message.

See also

setMessage()

9.39 nvcaffeparser1::IBinaryProtoBlob Class Reference

149

9.38.3.2

setMessage()

void nvinfer1::IAssertionLayer::setMessage (

char const ∗ message ) [inline], [noexcept]

Set the message to print if the assertion fails.

The name is used in error diagnostics. This method copies the message string.

See also

getMessage()

9.38.4

Member Data Documentation

9.38.4.1

mImpl

apiv::VAssertionLayer∗ nvinfer1::IAssertionLayer::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.39

nvcaffeparser1::IBinaryProtoBlob Class Reference

Object used to store and query data extracted from a binaryproto file using the ICaffeParser.

#include <NvCaffeParser.h>

Public Member Functions

• virtual void const ∗ getData () noexcept=0

• virtual nvinfer1::Dims4 getDimensions () noexcept=0

• virtual nvinfer1::DataType getDataType () noexcept=0

• virtual TRT DEPRECATED void destroy () noexcept=0

• virtual ∼IBinaryProtoBlob () noexcept=default

9.39.1

Detailed Description

Object used to store and query data extracted from a binaryproto file using the ICaffeParser.

See also

nvcaffeparser1::ICaffeParser

150

Class Documentation

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.39.2

Constructor & Destructor Documentation

9.39.2.1

∼IBinaryProtoBlob()

virtual nvcaffeparser1::IBinaryProtoBlob::∼IBinaryProtoBlob ( ) [virtual], [default], [noexcept]

9.39.3

Member Function Documentation

9.39.3.1

destroy()

virtual TRT DEPRECATED void nvcaffeparser1::IBinaryProtoBlob::destroy ( ) [pure virtual], [noexcept]

Deprecated Deprecated in TensorRT 8.0. Superseded by delete.

Warning

Calling destroy on a managed pointer will result in a double-free error.

9.39.3.2

getData()

virtual void const ∗ nvcaffeparser1::IBinaryProtoBlob::getData ( ) [pure virtual], [noexcept]

9.39.3.3

getDataType()

virtual nvinfer1::DataType nvcaffeparser1::IBinaryProtoBlob::getDataType ( ) [pure virtual],

[noexcept]

9.40 nvcaffeparser1::IBlobNameToTensor Class Reference

151

9.39.3.4

getDimensions()

virtual nvinfer1::Dims4 nvcaffeparser1::IBinaryProtoBlob::getDimensions ( ) [pure virtual], [noexcept]

The documentation for this class was generated from the following file:

• NvCaffeParser.h

9.40

nvcaffeparser1::IBlobNameToTensor Class Reference

Object used to store and query Tensors after they have been extracted from a Caffe model using the ICaffeParser.

#include <NvCaffeParser.h>

Public Member Functions

• virtual nvinfer1::ITensor ∗ find (char const ∗name) const noexcept=0

Given a blob name, returns a pointer to a ITensor object.

Protected Member Functions

• virtual ∼IBlobNameToTensor ()

9.40.1

Detailed Description

Object used to store and query Tensors after they have been extracted from a Caffe model using the ICaffeParser.

Note

The lifetime of IBlobNameToTensor is the same as the lifetime of its parent ICaffeParser.

See also

nvcaffeparser1::ICaffeParser

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.40.2

Constructor & Destructor Documentation

152

Class Documentation

9.40.2.1

∼IBlobNameToTensor()

virtual nvcaffeparser1::IBlobNameToTensor::∼IBlobNameToTensor ( ) [inline], [protected], [virtual]

9.40.3

Member Function Documentation

9.40.3.1

find()

virtual nvinfer1::ITensor ∗ nvcaffeparser1::IBlobNameToTensor::find (

char const ∗ name ) const [pure virtual], [noexcept]

Given a blob name, returns a pointer to a ITensor object.

Parameters

name

Caffe blob name for which the user wants the corresponding ITensor.

Returns

ITensor∗ corresponding to the queried name. If no such ITensor exists, then nullptr is returned.

The documentation for this class was generated from the following file:

• NvCaffeParser.h

9.41

nvinfer1::IBuilder Class Reference

Builds an engine from a network definition.

#include <NvInfer.h>

Inheritance diagram for nvinfer1::IBuilder:

nvinfer1::INoCopy

nvinfer1::IBuilder

9.41 nvinfer1::IBuilder Class Reference

153

Public Member Functions

•

virtual ∼IBuilder () noexcept=default

•

TRT DEPRECATED void setMaxBatchSize (int32 t batchSize) noexcept

Set the maximum batch size. This has no effect for networks created with explicit batch dimension mode.

•

TRT DEPRECATED int32 t getMaxBatchSize () const noexcept

Get the maximum batch size.

•

bool platformHasFastFp16 () const noexcept

Determine whether the platform has fast native fp16.

•

bool platformHasFastInt8 () const noexcept

Determine whether the platform has fast native int8.

•

TRT DEPRECATED void destroy () noexcept

Destroy this object.

•

int32 t getMaxDLABatchSize () const noexcept

Get the maximum batch size DLA can support. For any tensor the total volume of index dimensions combined(dimensions

other than CHW) with the requested batch size should not exceed the value returned by this function.

•

int32 t getNbDLACores () const noexcept

Return the number of DLA engines available to this builder.

•

void setGpuAllocator (IGpuAllocator ∗allocator) noexcept

Set the GPU allocator.

•

nvinfer1::IBuilderConfig ∗ createBuilderConfig () noexcept

Create a builder configuration object.

•

TRT DEPRECATED nvinfer1::ICudaEngine ∗ buildEngineWithConfig (INetworkDefinition &network,

IBuilderConfig &config) noexcept

Builds an engine for the given INetworkDefinition and given IBuilderConfig.

•

nvinfer1::INetworkDefinition ∗ createNetworkV2 (NetworkDefinitionCreationFlags flags) noexcept

Create a network definition object.

•

nvinfer1::IOptimizationProfile ∗ createOptimizationProfile () noexcept

Create a new optimization profile.

•

void setErrorRecorder (IErrorRecorder ∗recorder) noexcept

Set the ErrorRecorder for this interface.

•

IErrorRecorder ∗ getErrorRecorder () const noexcept

get the ErrorRecorder assigned to this interface.

•

void reset () noexcept

Resets the builder state to default values.

•

bool platformHasTf32 () const noexcept

Determine whether the platform has TF32 support.

•

nvinfer1::IHostMemory ∗ buildSerializedNetwork (INetworkDefinition &network, IBuilderConfig &config)

noexcept

Builds and serializes a network for the given INetworkDefinition and IBuilderConfig.

•

bool isNetworkSupported (INetworkDefinition const &network, IBuilderConfig const &config) const noexcept

Checks that a network is within the scope of the IBuilderConfig settings.

•

ILogger ∗ getLogger () const noexcept

get the logger with which the builder was created

•

bool setMaxThreads (int32 t maxThreads) noexcept

Set the maximum number of threads.

•

int32 t getMaxThreads () const noexcept

get the maximum number of threads that can be used by the builder.

•

IPluginRegistry & getPluginRegistry () noexcept

get the local plugin registry that can be used by the builder.

154

Class Documentation

Protected Attributes

• apiv::VBuilder ∗ mImpl

Additional Inherited Members

9.41.1

Detailed Description

Builds an engine from a network definition.

Warning

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

9.41.2

Constructor & Destructor Documentation

9.41.2.1

∼IBuilder()

virtual nvinfer1::IBuilder::∼IBuilder ( ) [virtual], [default], [noexcept]

9.41.3

Member Function Documentation

9.41.3.1

buildEngineWithConfig()

TRT DEPRECATED nvinfer1::ICudaEngine ∗ nvinfer1::IBuilder::buildEngineWithConfig (

INetworkDefinition & network,

IBuilderConfig & config ) [inline], [noexcept]

Builds an engine for the given INetworkDefinition and given IBuilderConfig.

It enables the builder to build multiple engines based on the same network definition, but with different builder config-

urations.

Note

This function will synchronize the cuda stream returned by config.getProfileStream() before return-

ing.

Deprecated Deprecated in TensorRT 8.0. Superseded by IBuilder::buildSerializedNetwork().

9.41.3.2

buildSerializedNetwork()

nvinfer1::IHostMemory ∗ nvinfer1::IBuilder::buildSerializedNetwork (

INetworkDefinition & network,

IBuilderConfig & config ) [inline], [noexcept]

Builds and serializes a network for the given INetworkDefinition and IBuilderConfig.

This function allows building and serialization of a network without creating an engine.

9.41 nvinfer1::IBuilder Class Reference

155

Parameters

network

Network definition.

config

Builder configuration.

Returns

A pointer to a IHostMemory object that contains a serialized network.

Note

This function will synchronize the cuda stream returned by config.getProfileStream() before return-

ing.

See also

INetworkDefinition, IBuilderConfig, IHostMemory

9.41.3.3

createBuilderConfig()

nvinfer1::IBuilderConfig ∗ nvinfer1::IBuilder::createBuilderConfig ( ) [inline], [noexcept]

Create a builder configuration object.

See also

IBuilderConfig

9.41.3.4

createNetworkV2()

nvinfer1::INetworkDefinition ∗ nvinfer1::IBuilder::createNetworkV2 (

NetworkDefinitionCreationFlags flags ) [inline], [noexcept]

Create a network definition object.

Creates a network definition object with immutable properties specified using the flags parameter. CreateNetworkV2

supports dynamic shapes and explicit batch dimensions when used with NetworkDefinitionCreationFlag::kEXPLICIT BATCH

flag. Creating a network without NetworkDefinitionCreationFlag::kEXPLICIT BATCH flag has been deprecated.

156

Class Documentation

Parameters

flags

Bitset of NetworkDefinitionCreationFlags specifying network properties combined with bitwise OR. e.g.,

1U << NetworkDefinitionCreationFlag::kEXPLICIT BATCH

See also

INetworkDefinition, NetworkDefinitionCreationFlags

9.41.3.5

createOptimizationProfile()

nvinfer1::IOptimizationProfile ∗ nvinfer1::IBuilder::createOptimizationProfile ( ) [inline],

[noexcept]

Create a new optimization profile.

If the network has any dynamic input tensors, the appropriate calls to setDimensions() must be made. Likewise, if there

are any shape input tensors, the appropriate calls to setShapeValues() are required. The builder retains ownership of

the created optimization profile and returns a raw pointer, i.e. the users must not attempt to delete the returned pointer.

See also

IOptimizationProfile

9.41.3.6

destroy()

TRT DEPRECATED void nvinfer1::IBuilder::destroy ( ) [inline], [noexcept]

Destroy this object.

Deprecated Deprecated in TensorRT 8.0. Superseded by delete.

Warning

Calling destroy on a managed pointer will result in a double-free error.

9.41 nvinfer1::IBuilder Class Reference

157

9.41.3.7

getErrorRecorder()

IErrorRecorder ∗ nvinfer1::IBuilder::getErrorRecorder ( ) const [inline], [noexcept]

get the ErrorRecorder assigned to this interface.

Retrieves the assigned error recorder object for the given class. A nullptr will be returned if setErrorRecorder has not

been called.

Returns

A pointer to the IErrorRecorder object that has been registered.

See also

setErrorRecorder()

9.41.3.8

getLogger()

ILogger ∗ nvinfer1::IBuilder::getLogger ( ) const [inline], [noexcept]

get the logger with which the builder was created

Returns

the logger

9.41.3.9

getMaxBatchSize()

TRT DEPRECATED int32 t nvinfer1::IBuilder::getMaxBatchSize ( ) const [inline], [noexcept]

Get the maximum batch size.

Returns

The maximum batch size.

Deprecated Deprecated in TensorRT 8.4.

See also

setMaxBatchSize()

getMaxDLABatchSize()

9.41.3.10 getMaxDLABatchSize()

int32 t nvinfer1::IBuilder::getMaxDLABatchSize ( ) const [inline], [noexcept]

Get the maximum batch size DLA can support. For any tensor the total volume of index dimensions com-

bined(dimensions other than CHW) with the requested batch size should not exceed the value returned by this function.

158

Class Documentation

Warning

getMaxDLABatchSize does not work with dynamic shapes.

9.41.3.11 getMaxThreads()

int32 t nvinfer1::IBuilder::getMaxThreads ( ) const [inline], [noexcept]

get the maximum number of threads that can be used by the builder.

Retrieves the maximum number of threads that can be used by the builder.

Returns

The maximum number of threads that can be used by the builder.

See also

setMaxThreads()

9.41.3.12 getNbDLACores()

int32 t nvinfer1::IBuilder::getNbDLACores ( ) const [inline], [noexcept]

Return the number of DLA engines available to this builder.

9.41.3.13 getPluginRegistry()

IPluginRegistry & nvinfer1::IBuilder::getPluginRegistry ( ) [inline], [noexcept]

get the local plugin registry that can be used by the builder.

Returns

The local plugin registry that can be used by the builder.

9.41.3.14 isNetworkSupported()

bool nvinfer1::IBuilder::isNetworkSupported (

INetworkDefinition const & network,

IBuilderConfig const & config ) const [inline], [noexcept]

Checks that a network is within the scope of the IBuilderConfig settings.

9.41 nvinfer1::IBuilder Class Reference

159

Parameters

network

The network definition to check for configuration compliance.

config

The configuration of the builder to use when checking network.

Given an INetworkDefinition, network, and an IBuilderConfig, config, check if the network falls within the con-

straints of the builder configuration based on the EngineCapability, BuilderFlag, and DeviceType. If the network is

within the constraints, then the function returns true, and false if a violation occurs. This function reports the conditions

that are violated to the registered ErrorRecorder.

Returns

True if network is within the scope of the restrictions specified by the builder config, false otherwise.

Note

This function will synchronize the cuda stream returned by config.getProfileStream() before return-

ing.

This function is only supported in NVIDIA Drive(R) products.

9.41.3.15 platformHasFastFp16()

bool nvinfer1::IBuilder::platformHasFastFp16 ( ) const [inline], [noexcept]

Determine whether the platform has fast native fp16.

9.41.3.16 platformHasFastInt8()

bool nvinfer1::IBuilder::platformHasFastInt8 ( ) const [inline], [noexcept]

Determine whether the platform has fast native int8.

9.41.3.17 platformHasTf32()

bool nvinfer1::IBuilder::platformHasTf32 ( ) const [inline], [noexcept]

Determine whether the platform has TF32 support.

160

Class Documentation

9.41.3.18 reset()

void nvinfer1::IBuilder::reset ( ) [inline], [noexcept]

Resets the builder state to default values.

9.41.3.19 setErrorRecorder()

void nvinfer1::IBuilder::setErrorRecorder (

IErrorRecorder ∗ recorder ) [inline], [noexcept]

Set the ErrorRecorder for this interface.

Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution. This function

will call incRefCount of the registered ErrorRecorder at least once. Setting recorder to nullptr unregisters the recorder

with the interface, resulting in a call to decRefCount if a recorder has been registered.

If an error recorder is not set, messages will be sent to the global log stream.

Parameters

recorder

The error recorder to register with this interface.

See also

getErrorRecorder()

9.41.3.20 setGpuAllocator()

void nvinfer1::IBuilder::setGpuAllocator (

IGpuAllocator ∗ allocator ) [inline], [noexcept]

Set the GPU allocator.

Parameters

allocator

Set the GPU allocator to be used by the builder. All GPU memory acquired will use this allocator. If

NULL is passed, the default allocator will be used.

Default: uses cudaMalloc/cudaFree.

9.41 nvinfer1::IBuilder Class Reference

161

Note

This allocator will be passed to any engines created via the builder; thus the lifetime of the allocator must span

the lifetime of those engines as well as that of the builder. If nullptr is passed, the default allocator will be used.

9.41.3.21 setMaxBatchSize()

TRT DEPRECATED void nvinfer1::IBuilder::setMaxBatchSize (

int32 t batchSize ) [inline], [noexcept]

Set the maximum batch size. This has no effect for networks created with explicit batch dimension mode.

Parameters

batchSize

The maximum batch size which can be used at execution time, and also the batch size for which the

engine will be optimized.

Deprecated Deprecated in TensorRT 8.4.

See also

getMaxBatchSize()

9.41.3.22 setMaxThreads()

bool nvinfer1::IBuilder::setMaxThreads (

int32 t maxThreads ) [inline], [noexcept]

Set the maximum number of threads.

Parameters

maxThreads

The maximum number of threads that can be used by the builder.

Returns

True if successful, false otherwise.

The default value is 1 and includes the current thread. A value greater than 1 permits TensorRT to use multi-threaded

algorithms. A value less than 1 triggers a kINVALID ARGUMENT error.

162

Class Documentation

9.41.4

Member Data Documentation

9.41.4.1

mImpl

apiv::VBuilder∗ nvinfer1::IBuilder::mImpl [protected]

The documentation for this class was generated from the following file:

• NvInfer.h

9.42

nvinfer1::IBuilderConfig Class Reference

Holds properties for configuring a builder to produce an engine.

#include <NvInfer.h>

Inheritance diagram for nvinfer1::IBuilderConfig:

nvinfer1::INoCopy

nvinfer1::IBuilderConfig

Public Member Functions

• virtual ∼IBuilderConfig () noexcept=default

• virtual TRT DEPRECATED void setMinTimingIterations (int32 t minTiming) noexcept

Set the number of minimization iterations used when timing layers.

• virtual TRT DEPRECATED int32 t getMinTimingIterations () const noexcept

Query the number of minimization iterations.

• virtual void setAvgTimingIterations (int32 t avgTiming) noexcept

Set the number of averaging iterations used when timing layers.

• int32 t getAvgTimingIterations () const noexcept

Query the number of averaging iterations.

• void setEngineCapability (EngineCapability capability) noexcept

Configure the builder to target specified EngineCapability flow.

• EngineCapability getEngineCapability () const noexcept

Query EngineCapability flow configured for the builder.

• void setInt8Calibrator (IInt8Calibrator ∗calibrator) noexcept

Set Int8 Calibration interface.

• IInt8Calibrator ∗ getInt8Calibrator () const noexcept

9.42 nvinfer1::IBuilderConfig Class Reference

163

Get Int8 Calibration interface.

•

TRT DEPRECATED void setMaxWorkspaceSize (std::size t workspaceSize) noexcept

Set the maximum workspace size.

•

TRT DEPRECATED std::size t getMaxWorkspaceSize () const noexcept

Get the maximum workspace size.

•

void setFlags (BuilderFlags builderFlags) noexcept

Set the build mode flags to turn on builder options for this network.

•

BuilderFlags getFlags () const noexcept

Get the build mode flags for this builder config. Defaults to 0.

•

void clearFlag (BuilderFlag builderFlag) noexcept

clear a single build mode flag.

•

void setFlag (BuilderFlag builderFlag) noexcept

Set a single build mode flag.

•

bool getFlag (BuilderFlag builderFlag) const noexcept

Returns true if the build mode flag is set.

•

void setDeviceType (ILayer const ∗layer, DeviceType deviceType) noexcept

Set the device that this layer must execute on.

•

DeviceType getDeviceType (ILayer const ∗layer) const noexcept

Get the device that this layer executes on.

•

bool isDeviceTypeSet (ILayer const ∗layer) const noexcept

whether the DeviceType has been explicitly set for this layer

•

void resetDeviceType (ILayer const ∗layer) noexcept

reset the DeviceType for this layer

•

bool canRunOnDLA (ILayer const ∗layer) const noexcept

Checks if a layer can run on DLA.

•

void setDLACore (int32 t dlaCore) noexcept

Sets the DLA core used by the network. Defaults to -1.

•

int32 t getDLACore () const noexcept

Get the DLA core that the engine executes on.

•

void setDefaultDeviceType (DeviceType deviceType) noexcept

Sets the default DeviceType to be used by the builder. It ensures that all the layers that can run on this device will run on

it, unless setDeviceType is used to override the default DeviceType for a layer.

•

DeviceType getDefaultDeviceType () const noexcept

Get the default DeviceType which was set by setDefaultDeviceType.

•

void reset () noexcept

Resets the builder configuration to defaults.

•

TRT DEPRECATED void destroy () noexcept

Delete this IBuilderConfig.

•

void setProfileStream (const cudaStream t stream) noexcept

Set the cuda stream that is used to profile this network.

•

cudaStream t getProfileStream () const noexcept

Get the cuda stream that is used to profile this network.

•

int32 t addOptimizationProfile (IOptimizationProfile const ∗profile) noexcept

Add an optimization profile.

•

int32 t getNbOptimizationProfiles () const noexcept

Get number of optimization profiles.

•

void setProfilingVerbosity (ProfilingVerbosity verbosity) noexcept

164

Class Documentation

Set verbosity level of layer information exposed in NVTX annotations and IEngineInspector.

•

ProfilingVerbosity getProfilingVerbosity () const noexcept

Get verbosity level of layer information exposed in NVTX annotations and IEngineInspector.

•

void setAlgorithmSelector (IAlgorithmSelector ∗selector) noexcept

Set Algorithm Selector.

•

IAlgorithmSelector ∗ getAlgorithmSelector () const noexcept

Get Algorithm Selector.

•

bool setCalibrationProfile (IOptimizationProfile const ∗profile) noexcept

Add a calibration profile.

•

IOptimizationProfile const ∗ getCalibrationProfile () noexcept

Get the current calibration profile.

•

void setQuantizationFlags (QuantizationFlags flags) noexcept

Set the quantization flags.

•

QuantizationFlags getQuantizationFlags () const noexcept

Get the quantization flags.

•

void clearQuantizationFlag (QuantizationFlag flag) noexcept

clear a quantization flag.

•

void setQuantizationFlag (QuantizationFlag flag) noexcept

Set a single quantization flag.

•

bool getQuantizationFlag (QuantizationFlag flag) const noexcept

Returns true if the quantization flag is set.

•

bool setTacticSources (TacticSources tacticSources) noexcept

Set tactic sources.

•

TacticSources getTacticSources () const noexcept

Get tactic sources.

•

nvinfer1::ITimingCache ∗ createTimingCache (void const ∗blob, std::size t size) const noexcept

Create timing cache.

•

bool setTimingCache (ITimingCache const &cache, bool ignoreMismatch) noexcept

Attach a timing cache to IBuilderConfig.

•

nvinfer1::ITimingCache const ∗ getTimingCache () const noexcept

Get the pointer to the timing cache from current IBuilderConfig.

•

void setMemoryPoolLimit (MemoryPoolType pool, std::size t poolSize) noexcept

Set the memory size for the memory pool.

•

std::size t getMemoryPoolLimit (MemoryPoolType pool) const noexcept

Get the memory size limit of the memory pool.

•

void setPreviewFeature (PreviewFeature feature, bool enable) noexcept

Enable or disable a specific preview feature.

•

bool getPreviewFeature (PreviewFeature feature) const noexcept

Get status of preview feature.

•

void setBuilderOptimizationLevel (int32 t level) noexcept

Set builder optimization level.

•

int32 t getBuilderOptimizationLevel () noexcept

Get builder optimization level.

•

void setHardwareCompatibilityLevel (HardwareCompatibilityLevel hardwareCompatibilityLevel) noexcept

Set the hardware compatibility level.

•

HardwareCompatibilityLevel getHardwareCompatibilityLevel () const noexcept

Get the hardware compatibility level.

9.42 nvinfer1::IBuilderConfig Class Reference

165

• void setPluginsToSerialize (char const ∗const ∗paths, int32 t nbPaths) noexcept

Set the plugin libraries to be serialized with version-compatible engines.

• char const ∗ getPluginToSerialize (int32 t index) const noexcept

Get the plugin library path to be serialized with version-compatible engines.

• int32 t getNbPluginsToSerialize () const noexcept

Get the number of plugin library paths to be serialized with version-compatible engines.

• void setMaxAuxStreams (int32 t nbStreams) noexcept

Set the maximum number of auxiliary streams that TRT is allowed to use.

• int32 t getMaxAuxStreams () const noexcept

Get the maximum number of auxiliary streams that TRT is allowed to use.

Protected Attributes

• apiv::VBuilderConfig ∗ mImpl

Additional Inherited Members

9.42.1

Detailed Description

Holds properties for configuring a builder to produce an engine.

See also

BuilderFlags

9.42.2

Constructor & Destructor Documentation

9.42.2.1

∼IBuilderConfig()

virtual nvinfer1::IBuilderConfig::∼IBuilderConfig ( ) [virtual], [default], [noexcept]

9.42.3

Member Function Documentation

9.42.3.1

addOptimizationProfile()

int32 t nvinfer1::IBuilderConfig::addOptimizationProfile (

IOptimizationProfile const ∗ profile ) [inline], [noexcept]

Add an optimization profile.

This function must be called at least once if the network has dynamic or shape input tensors. This function may be

called at most once when building a refittable engine, as more than a single optimization profile are not supported for

refittable engines.

166

Class Documentation

Parameters

profile

The new optimization profile, which must satisfy profile->isValid() == true

Returns

The index of the optimization profile (starting from 0) if the input is valid, or -1 if the input is not valid.

9.42.3.2

canRunOnDLA()

bool nvinfer1::IBuilderConfig::canRunOnDLA (

ILayer const ∗ layer ) const [inline], [noexcept]

Checks if a layer can run on DLA.

Returns

status true if the layer can on DLA else returns false.

9.42.3.3

clearFlag()

void nvinfer1::IBuilderConfig::clearFlag (

BuilderFlag builderFlag ) [inline], [noexcept]

clear a single build mode flag.

clears the builder mode flag from the enabled flags.

See also

setFlags()

9.42.3.4

clearQuantizationFlag()

void nvinfer1::IBuilderConfig::clearQuantizationFlag (

QuantizationFlag flag ) [inline], [noexcept]

clear a quantization flag.

Clears the quantization flag from the enabled quantization flags.

See also

setQuantizationFlags()

9.42 nvinfer1::IBuilderConfig Class Reference

167

9.42.3.5

createTimingCache()

nvinfer1::ITimingCache ∗ nvinfer1::IBuilderConfig::createTimingCache (

void const ∗ blob,

std::size t size ) const [inline], [noexcept]

Create timing cache.

Create ITimingCache instance from serialized raw data. The created timing cache doesn’t belong to a specific

IBuilderConfig. It can be shared by multiple builder instances. Call setTimingCache() before launching a builder

to attach cache to builder instance.

Parameters

blob

A pointer to the raw data that contains serialized timing cache

size

The size in bytes of the serialized timing cache. Size 0 means create a new cache from scratch

See also

setTimingCache

Returns

the pointer to ITimingCache created

9.42.3.6

destroy()

TRT DEPRECATED void nvinfer1::IBuilderConfig::destroy ( ) [inline], [noexcept]

Delete this IBuilderConfig.

De-allocates any internally allocated memory.

Deprecated Deprecated in TensorRT 8.0. Superseded by delete.

Warning

Calling destroy on a managed pointer will result in a double-free error.

168

Class Documentation

9.42.3.7

getAlgorithmSelector()

IAlgorithmSelector ∗ nvinfer1::IBuilderConfig::getAlgorithmSelector ( ) const [inline], [noexcept]

Get Algorithm Selector.

9.42.3.8

getAvgTimingIterations()

int32 t nvinfer1::IBuilderConfig::getAvgTimingIterations ( ) const [inline], [noexcept]

Query the number of averaging iterations.

By default the number of averaging iterations is 1.

See also

setAvgTimingIterations()

9.42.3.9

getBuilderOptimizationLevel()

int32 t nvinfer1::IBuilderConfig::getBuilderOptimizationLevel ( ) [inline], [noexcept]

Get builder optimization level.

Returns

the current builder optimization level

See also

setBuilderOptimizationLevel

9.42.3.10 getCalibrationProfile()

IOptimizationProfile const ∗ nvinfer1::IBuilderConfig::getCalibrationProfile ( ) [inline], [noexcept]

Get the current calibration profile.

Returns

A pointer to the current calibration profile or nullptr if calibration profile is unset.

9.42 nvinfer1::IBuilderConfig Class Reference

169

9.42.3.11 getDefaultDeviceType()

DeviceType nvinfer1::IBuilderConfig::getDefaultDeviceType ( ) const [inline], [noexcept]

Get the default DeviceType which was set by setDefaultDeviceType.

By default it returns DeviceType::kGPU.

9.42.3.12 getDeviceType()

DeviceType nvinfer1::IBuilderConfig::getDeviceType (

ILayer const ∗ layer ) const [inline], [noexcept]

Get the device that this layer executes on.

Returns

Returns DeviceType of the layer.

9.42.3.13 getDLACore()

int32 t nvinfer1::IBuilderConfig::getDLACore ( ) const [inline], [noexcept]

Get the DLA core that the engine executes on.

Returns

assigned DLA core or -1 for DLA not present or unset.

9.42.3.14 getEngineCapability()

EngineCapability nvinfer1::IBuilderConfig::getEngineCapability ( ) const [inline], [noexcept]

Query EngineCapability flow configured for the builder.

By default it returns EngineCapability::kSTANDARD.

See also

setEngineCapability()

170

Class Documentation

9.42.3.15 getFlag()

bool nvinfer1::IBuilderConfig::getFlag (

BuilderFlag builderFlag ) const [inline], [noexcept]

Returns true if the build mode flag is set.

See also

getFlags()

Returns

True if flag is set, false if unset.

9.42.3.16 getFlags()

BuilderFlags nvinfer1::IBuilderConfig::getFlags ( ) const [inline], [noexcept]

Get the build mode flags for this builder config. Defaults to 0.

Returns

The build options as a bitmask.

See also

setFlags()

9.42.3.17 getHardwareCompatibilityLevel()

HardwareCompatibilityLevel nvinfer1::IBuilderConfig::getHardwareCompatibilityLevel ( ) const [inline],

[noexcept]

Get the hardware compatibility level.

Returns

hardwareCompatibilityLevel The level of hardware compatibility.

See also

setHardwareCompatiblityLevel()

9.42 nvinfer1::IBuilderConfig Class Reference

171

9.42.3.18 getInt8Calibrator()

IInt8Calibrator ∗ nvinfer1::IBuilderConfig::getInt8Calibrator ( ) const [inline], [noexcept]

Get Int8 Calibration interface.

9.42.3.19 getMaxAuxStreams()

int32 t nvinfer1::IBuilderConfig::getMaxAuxStreams ( ) const [inline], [noexcept]

Get the maximum number of auxiliary streams that TRT is allowed to use.

See also

setMaxAuxStreams()

9.42.3.20 getMaxWorkspaceSize()

TRT DEPRECATED std::size t nvinfer1::IBuilderConfig::getMaxWorkspaceSize ( ) const [inline], [noexcept]

Get the maximum workspace size.

By default the workspace size is the size of total global memory in the device.

Returns

The maximum workspace size.

See also

setMaxWorkspaceSize()

Deprecated Deprecated in TensorRT 8.3. Superseded by IBuilderConfig::getMemoryPoolLimit() with MemoryPoolType::kWORKSPA

9.42.3.21 getMemoryPoolLimit()

std::size t nvinfer1::IBuilderConfig::getMemoryPoolLimit (

MemoryPoolType pool ) const [inline], [noexcept]

Get the memory size limit of the memory pool.

Retrieve the memory size limit of the corresponding pool in bytes. If setMemoryPoolLimit for the pool has not been

called, this returns the default value used by TensorRT. This default value is not necessarily the maximum possible

value for that configuration.

172

Class Documentation

Parameters

pool

The memory pool to get the limit for.

Returns

The size of the memory limit, in bytes, for the corresponding pool.

See also

setMemoryPoolLimit

9.42.3.22 getMinTimingIterations()

virtual TRT DEPRECATED int32 t nvinfer1::IBuilderConfig::getMinTimingIterations ( ) const [inline],

[virtual], [noexcept]

Query the number of minimization iterations.

By default the minimum number of iterations is 1.

See also

setMinTimingIterations()

Deprecated Deprecated in TensorRT 8.4. Superseded by getAvgTimingIterations().

9.42.3.23 getNbOptimizationProfiles()

int32 t nvinfer1::IBuilderConfig::getNbOptimizationProfiles ( ) const [inline], [noexcept]

Get number of optimization profiles.

This is one higher than the index of the last optimization profile that has be defined (or zero, if none has been defined

yet).

Returns

The number of the optimization profiles.

9.42 nvinfer1::IBuilderConfig Class Reference

173

9.42.3.24 getNbPluginsToSerialize()

int32 t nvinfer1::IBuilderConfig::getNbPluginsToSerialize ( ) const [inline], [noexcept]

Get the number of plugin library paths to be serialized with version-compatible engines.

Returns

The number of paths.

9.42.3.25 getPluginToSerialize()

char const ∗ nvinfer1::IBuilderConfig::getPluginToSerialize (

int32 t index ) const [inline], [noexcept]

Get the plugin library path to be serialized with version-compatible engines.

Parameters

index

Index of the plugin library path in the list. Should be in the range [0,

getNbPluginsToSerialize()).

Returns

The path to the plugin library.

9.42.3.26 getPreviewFeature()

bool nvinfer1::IBuilderConfig::getPreviewFeature (

PreviewFeature feature ) const [inline], [noexcept]

Get status of preview feature.

Parameters

feature

the feature to query

Returns

true if the feature is enabled, false otherwise

174

Class Documentation

See also

PreviewFeature, setPreviewFeature

9.42.3.27 getProfileStream()

cudaStream t nvinfer1::IBuilderConfig::getProfileStream ( ) const [inline], [noexcept]

Get the cuda stream that is used to profile this network.

Returns

The cuda stream set by setProfileStream, nullptr if setProfileStream has not been called.

See also

setProfileStream()

9.42.3.28 getProfilingVerbosity()

ProfilingVerbosity nvinfer1::IBuilderConfig::getProfilingVerbosity ( ) const [inline], [noexcept]

Get verbosity level of layer information exposed in NVTX annotations and IEngineInspector.

Get the current setting of verbosity level of layer information exposed in NVTX annotations and IEngineInspector.

Default value is ProfilingVerbosity::kLAYER NAMES ONLY.

See also

ProfilingVerbosity, setProfilingVerbosity(), IEngineInspector

9.42.3.29 getQuantizationFlag()

bool nvinfer1::IBuilderConfig::getQuantizationFlag (

QuantizationFlag flag ) const [inline], [noexcept]

Returns true if the quantization flag is set.

See also

getQuantizationFlags()

Returns

True if quantization flag is set, false if unset.

9.42 nvinfer1::IBuilderConfig Class Reference

175

9.42.3.30 getQuantizationFlags()

QuantizationFlags nvinfer1::IBuilderConfig::getQuantizationFlags ( ) const [inline], [noexcept]

Get the quantization flags.

Returns

The quantization flags as a bitmask.

See also

setQuantizationFlag()

9.42.3.31 getTacticSources()

TacticSources nvinfer1::IBuilderConfig::getTacticSources ( ) const [inline], [noexcept]

Get tactic sources.

Get the tactic sources currently set in the engine build configuration.

See also

setTacticSources()

Returns

tactic sources

9.42.3.32 getTimingCache()

nvinfer1::ITimingCache const ∗ nvinfer1::IBuilderConfig::getTimingCache ( ) const [inline], [noexcept]

Get the pointer to the timing cache from current IBuilderConfig.

Returns

pointer to the timing cache used in current IBuilderConfig

176

Class Documentation

9.42.3.33 isDeviceTypeSet()

bool nvinfer1::IBuilderConfig::isDeviceTypeSet (

ILayer const ∗ layer ) const [inline], [noexcept]

whether the DeviceType has been explicitly set for this layer

Returns

true if device type is not default

See also

setDeviceType() getDeviceType() resetDeviceType()

9.42.3.34 reset()

void nvinfer1::IBuilderConfig::reset ( ) [inline], [noexcept]

Resets the builder configuration to defaults.

Useful for initializing a builder config object to its original state.

9.42.3.35 resetDeviceType()

void nvinfer1::IBuilderConfig::resetDeviceType (

ILayer const ∗ layer ) [inline], [noexcept]

reset the DeviceType for this layer

See also

setDeviceType() getDeviceType() isDeviceTypeSet()

9.42.3.36 setAlgorithmSelector()

void nvinfer1::IBuilderConfig::setAlgorithmSelector (

IAlgorithmSelector ∗ selector ) [inline], [noexcept]

Set Algorithm Selector.

9.42 nvinfer1::IBuilderConfig Class Reference

177

Parameters

selector

The algorithm selector to be set in the build config.

9.42.3.37 setAvgTimingIterations()

virtual void nvinfer1::IBuilderConfig::setAvgTimingIterations (

int32 t avgTiming ) [inline], [virtual], [noexcept]

Set the number of averaging iterations used when timing layers.

When timing layers, the builder minimizes over a set of average times for layer execution. This parameter controls the

number of iterations used in averaging.

See also

getAvgTimingIterations()

9.42.3.38 setBuilderOptimizationLevel()

void nvinfer1::IBuilderConfig::setBuilderOptimizationLevel (

int32 t level ) [inline], [noexcept]

Set builder optimization level.

Set the builder optimization level. Setting a higher optimization level allows the optimizer to spend more time searching

for optimization opportunities. The resulting engine may have better performance compared to an engine built with a

lower optimization level.

The default optimization level is 3. Valid values include integers from 0 to the maximum optimization level, which is

currently 5. Setting it to greater than the maximum level results in behavior identical to the maximum level.

Parameters

level

The optimization level to set to. Must be non-negative.

See also

getBuilderOptimizationLevel

178

Class Documentation

9.42.3.39 setCalibrationProfile()

bool nvinfer1::IBuilderConfig::setCalibrationProfile (

IOptimizationProfile const ∗ profile ) [inline], [noexcept]

Add a calibration profile.

Calibration optimization profile must be set if int8 calibration is used to set scales for a network with runtime dimen-

sions.

Parameters

profile

The new calibration profile, which must satisfy profile->isValid() == true or be nullptr. MIN and MAX

values will be overwritten by kOPT.

Returns

True if the calibration profile was set correctly.

9.42.3.40 setDefaultDeviceType()

void nvinfer1::IBuilderConfig::setDefaultDeviceType (

DeviceType deviceType ) [inline], [noexcept]

Sets the default DeviceType to be used by the builder. It ensures that all the layers that can run on this device will run

on it, unless setDeviceType is used to override the default DeviceType for a layer.

See also

getDefaultDeviceType()

9.42.3.41 setDeviceType()

void nvinfer1::IBuilderConfig::setDeviceType (

ILayer const ∗ layer,

DeviceType deviceType ) [inline], [noexcept]

Set the device that this layer must execute on.

Parameters

layer

which layer to execute.

deviceType

that this layer must execute on. If DeviceType is not set or is reset, TensorRT will use the default

DeviceType set in the builder.

9.42 nvinfer1::IBuilderConfig Class Reference

179

Note

The device type for a layer must be compatible with the safety flow (if specified). For example a layer cannot be

marked for DLA execution while the builder is configured for kSAFE GPU.

See also

getDeviceType()

9.42.3.42 setDLACore()

void nvinfer1::IBuilderConfig::setDLACore (

int32 t dlaCore ) [inline], [noexcept]

Sets the DLA core used by the network. Defaults to -1.

Parameters

dlaCore

The DLA core to execute the engine on, in the range [0,getNbDlaCores()).

This function is used to specify which DLA core to use via indexing, if multiple DLA cores are available.

Warning

if getNbDLACores() returns 0, then this function does nothing.

See also

IRuntime::setDLACore() getDLACore()

9.42.3.43 setEngineCapability()

void nvinfer1::IBuilderConfig::setEngineCapability (

EngineCapability capability ) [inline], [noexcept]

Configure the builder to target specified EngineCapability flow.

The flow means a sequence of API calls that allow an application to set up a runtime, engine, and execution context in

order to run inference.

The supported flows are specified in the EngineCapability enum.

180

Class Documentation

9.42.3.44 setFlag()

void nvinfer1::IBuilderConfig::setFlag (

BuilderFlag builderFlag ) [inline], [noexcept]

Set a single build mode flag.

Add the input builder mode flag to the already enabled flags.

See also

setFlags()

9.42.3.45 setFlags()

void nvinfer1::IBuilderConfig::setFlags (

BuilderFlags builderFlags ) [inline], [noexcept]

Set the build mode flags to turn on builder options for this network.

The flags are listed in the BuilderFlags enum. The flags set configuration options to build the network.

Parameters

builderFlags

The build option for an engine.

Note

This function will override the previous set flags, rather than bitwise ORing the new flag.

See also

getFlags()

9.42.3.46 setHardwareCompatibilityLevel()

void nvinfer1::IBuilderConfig::setHardwareCompatibilityLevel (

HardwareCompatibilityLevel hardwareCompatibilityLevel ) [inline], [noexcept]

Set the hardware compatibility level.

Hardware compatibility allows an engine to run on GPU architectures other than that of the GPU where the engine was

built.

The default hardware compatibility level is HardwareCompatibilityLevel::kNONE.

9.42 nvinfer1::IBuilderConfig Class Reference

181

Parameters

hardwareCompatibilityLevel

The level of hardware compatibility.

9.42.3.47 setInt8Calibrator()

void nvinfer1::IBuilderConfig::setInt8Calibrator (

IInt8Calibrator ∗ calibrator ) [inline], [noexcept]

Set Int8 Calibration interface.

The calibrator is to minimize the information loss during the INT8 quantization process.

9.42.3.48 setMaxAuxStreams()

void nvinfer1::IBuilderConfig::setMaxAuxStreams (

int32 t nbStreams ) [inline], [noexcept]

Set the maximum number of auxiliary streams that TRT is allowed to use.

If the network contains operators that can run in parallel, TRT can execute them using auxiliary streams in addition to

the one provided to the IExecutionContext::enqueueV3() call.

The default maximum number of auxiliary streams is determined by the heuristics in TensorRT on whether enabling

multi-stream would improve the performance. This behavior can be overridden by calling this API to set the maximum

number of auxiliary streams explicitly. Set this to 0 to enforce single-stream inference.

The resulting engine may use fewer auxiliary streams than the maximum if the network does not contain enough

parallelism or if TensorRT determines that using more auxiliary streams does not help improve the performance.

Note

Allowing more auxiliary streams does not always give better performance since there will be synchronizations

overhead between streams. Using CUDA graphs at runtime can help reduce the overhead caused by cross-stream

synchronizations.

Using more auxiliary leads to more memory usage at runtime since some activation memory blocks will not be

able to be reused.

Parameters

nbStreams

The maximum number of auxiliary streams that TRT is allowed to use.

NVIDIA TensorRT 8.6.11 API Reference for DRIVE OS (May 2023) - page 3