NVIDIA TensorRT 8.6.11 API Reference for DRIVE OS (May 2023)

Namespace Documentation

•

class IDequantizeLayer

A Dequantize layer in a network definition.

•

class IDimensionExpr

•

class IEinsumLayer

An Einsum layer in a network.

•

class IElementWiseLayer

A elementwise layer in a network definition.

•

class IEngineInspector

An engine inspector which prints out the layer information of an engine or an execution context.

•

class IErrorRecorder

Reference counted application-implemented error reporting interface for TensorRT objects.

•

class IExecutionContext

Context for executing inference using an engine, with functionally unsafe features.

•

class IExprBuilder

•

class IFillLayer

Generate an output tensor with specified mode.

•

class IFullyConnectedLayer

A fully connected layer in a network definition. This layer expects an input tensor of three or more non-batch dimensions.

The input is automatically reshaped into an MxV tensor X, where V is a product of the last three dimensions and M is a

product of the remaining dimensions (where the product over 0 dimensions is defined as 1). For example:

•

class IGatherLayer

A Gather layer in a network definition. Supports several kinds of gathering.

•

class IGpuAllocator

Application-implemented class for controlling allocation on the GPU.

•

class IGridSampleLayer

A GridSample layer in a network definition.

•

class IHostMemory

Class to handle library allocated memory that is accessible to the user.

•

class IIdentityLayer

A layer that represents the identity function.

•

class IIfConditional

•

class IIfConditionalBoundaryLayer

•

class IIfConditionalInputLayer

•

class IIfConditionalOutputLayer

•

class IInt8Calibrator

Application-implemented interface for calibration.

•

class IInt8EntropyCalibrator

•

class IInt8EntropyCalibrator2

•

class IInt8LegacyCalibrator

•

class IInt8MinMaxCalibrator

•

class IIteratorLayer

•

class ILayer

Base class for all layer classes in a network definition.

•

class ILogger

Application-implemented logging interface for the builder, refitter and runtime.

•

class ILoggerFinder

A virtual base class to find a logger. Allows a plugin to find an instance of a logger if it needs to emit a log message. A

pointer to an instance of this class is passed to a plugin shared library on initialization when that plugin is serialized as

part of a version-compatible plan. See the plugin chapter in the developer guide for details.

8.2 nvinfer1 Namespace Reference

•

class ILoop

•

class ILoopBoundaryLayer

•

class ILoopOutputLayer

•

class ILRNLayer

A LRN layer in a network definition.

•

class IMatrixMultiplyLayer

Layer that represents a Matrix Multiplication.

•

class INetworkDefinition

A network definition for input to the builder.

•

class INMSLayer

A non-maximum suppression layer in a network definition.

•

class INoCopy

Forward declaration of IEngineInspector for use by other interfaces.

•

class INonZeroLayer

•

class INormalizationLayer

A normalization layer in a network definition.

•

class IOneHotLayer

A OneHot layer in a network definition.

•

class IOptimizationProfile

Optimization profile for dynamic input dimensions and shape tensors.

•

class IOutputAllocator

Callback from ExecutionContext::enqueueV3()

•

class IPaddingLayer

Layer that represents a padding operation.

•

class IParametricReLULayer

Layer that represents a parametric ReLU operation.

•

class IPluginCreator

Plugin creator class for user implemented layers.

•

class IPluginRegistry

Single registration point for all plugins in an application. It is used to find plugin implementations during engine dese-

rialization. Internally, the plugin registry is considered to be a singleton so all plugins in an application are part of the

same global registry. Note that the plugin registry is only supported for plugins of type IPluginV2 and should also have

a corresponding IPluginCreator implementation.

•

class IPluginV2

Plugin class for user-implemented layers.

•

class IPluginV2DynamicExt

•

class IPluginV2Ext

Plugin class for user-implemented layers.

•

class IPluginV2IOExt

Plugin class for user-implemented layers.

•

class IPluginV2Layer

Layer type for pluginV2.

•

class IPoolingLayer

A Pooling layer in a network definition.

•

class IProfiler

Application-implemented interface for profiling.

•

class IQuantizeLayer

A Quantize layer in a network definition.

Namespace Documentation

•

class IRaggedSoftMaxLayer

A RaggedSoftmax layer in a network definition.

•

class IRecurrenceLayer

•

class IReduceLayer

Layer that represents a reduction across a non-bool tensor.

•

class IRefitter

Updates weights in an engine.

•

class IResizeLayer

A resize layer in a network definition.

•

class IReverseSequenceLayer

A ReverseSequence layer in a network definition.

•

class IRNNv2Layer

An RNN layer in a network definition, version 2.

•

class IRuntime

Allows a serialized functionally unsafe engine to be deserialized.

•

class IScaleLayer

A Scale layer in a network definition.

•

class IScatterLayer

A scatter layer in a network definition. Supports several kinds of scattering.

•

class ISelectLayer

•

class IShapeLayer

Layer type for getting shape of a tensor.

•

class IShuffleLayer

Layer type for shuffling data.

•

class ISliceLayer

Slices an input tensor into an output tensor based on the offset and strides.

•

class ISoftMaxLayer

A Softmax layer in a network definition.

•

class ITensor

A tensor in a network definition.

•

class ITimingCache

Class to handle tactic timing info collected from builder.

•

class ITopKLayer

Layer that represents a TopK reduction.

•

class ITripLimitLayer

•

class IUnaryLayer

Layer that represents an unary operation.

•

struct Permutation

•

class PluginField

Structure containing plugin attribute field names and associated data This information can be parsed to decode necessary

plugin metadata.

•

struct PluginFieldCollection

Plugin field collection struct.

•

class PluginRegistrar

Register the plugin creator to the registry The static registry object will be instantiated when the plugin library is loaded.

This static object will register all creators available in the library to the registry.

•

struct PluginTensorDesc

Fields that a plugin might see for an input or output.

•

class Weights

An array of weights used as a layer parameter.

8.2 nvinfer1 Namespace Reference

Typedefs

• using TensorFormats = uint32 t

It is capable of representing one or more TensorFormat by binary OR operations, e.g., 1U << TensorFormat::kCHW4 |

1U << TensorFormat::kCHW32.

• using SliceMode = SampleMode

• using ResizeMode = InterpolationMode

• using QuantizationFlags = uint32 t

Represents one or more QuantizationFlag values using binary OR operations.

• using BuilderFlags = uint32 t

Represents one or more BuilderFlag values using binary OR operations, e.g., 1U << BuilderFlag::kFP16 | 1U <<

BuilderFlag::kDEBUG.

• using NetworkDefinitionCreationFlags = uint32 t

Represents one or more NetworkDefinitionCreationFlag flags using binary OR operations.

e.g., 1U

NetworkDefinitionCreationFlag::kEXPLICIT BATCH.

• using TempfileControlFlags = uint32 t

Represents a collection of one or more TempfileControlFlag values combined using bitwise-OR operations.

• using TacticSources = uint32 t

Represents a collection of one or more TacticSource values combine using bitwise-OR operations.

• using char t = char

char t is the type used by TensorRT to represent all valid characters.

• using AsciiChar = char t

• using Dims = Dims32

• using AllocatorFlags = uint32 t

• using PluginFormat = TensorFormat

PluginFormat is reserved for backward compatibility.

Enumerations

• enum class LayerType : int32 t {

kCONVOLUTION = 0 , kFULLY CONNECTED = 1 , kACTIVATION = 2 , kPOOLING = 3 ,

kLRN = 4 , kSCALE = 5 , kSOFTMAX = 6 , kDECONVOLUTION = 7 ,

kCONCATENATION = 8 , kELEMENTWISE = 9 , kPLUGIN = 10 , kUNARY = 11 ,

kPADDING = 12 , kSHUFFLE = 13 , kREDUCE = 14 , kTOPK = 15 ,

kGATHER = 16 , kMATRIX MULTIPLY = 17 , kRAGGED SOFTMAX = 18 , kCONSTANT = 19 ,

kRNN V2 = 20 , kIDENTITY = 21 , kPLUGIN V2 = 22 , kSLICE = 23 ,

kSHAPE = 24 , kPARAMETRIC RELU = 25 , kRESIZE = 26 , kTRIP LIMIT = 27 ,

kRECURRENCE = 28 , kITERATOR = 29 , kLOOP OUTPUT = 30 , kSELECT = 31 ,

kFILL = 32 , kQUANTIZE = 33 , kDEQUANTIZE = 34 , kCONDITION = 35 ,

kCONDITIONAL INPUT = 36 , kCONDITIONAL OUTPUT = 37 , kSCATTER = 38 , kEINSUM = 39 ,

kASSERTION = 40 , kONE HOT = 41 , kNON ZERO = 42 , kGRID SAMPLE = 43 ,

kNMS = 44 , kREVERSE SEQUENCE = 45 , kNORMALIZATION = 46 , kCAST = 47 }

The type values of layer classes.

• enum class ActivationType : int32 t {

kRELU = 0 , kSIGMOID = 1 , kTANH = 2 , kLEAKY RELU = 3 ,

kELU = 4 , kSELU = 5 , kSOFTSIGN = 6 , kSOFTPLUS = 7 ,

kCLIP = 8 , kHARD SIGMOID = 9 , kSCALED TANH = 10 , kTHRESHOLDED RELU = 11 }

Enumerates the types of activation to perform in an activation layer.

Namespace Documentation

•

enum class PaddingMode : int32 t {

kEXPLICIT ROUND DOWN = 0 , kEXPLICIT ROUND UP = 1 , kSAME UPPER = 2 , kSAME LOWER =

kCAFFE ROUND DOWN = 4 , kCAFFE ROUND UP = 5 }

Enumerates the modes of padding to perform in convolution, deconvolution and pooling layer, padding mode takes

precedence if setPaddingMode() and setPrePadding() are also used.

•

enum class PoolingType : int32 t { kMAX = 0 , kAVERAGE = 1 , kMAX AVERAGE BLEND = 2 }

The type of pooling to perform in a pooling layer.

•

enum class ScaleMode : int32 t { kUNIFORM = 0 , kCHANNEL = 1 , kELEMENTWISE = 2 }

Controls how shift, scale and power are applied in a Scale layer.

•

enum class ElementWiseOperation : int32 t {

kSUM = 0 , kPROD = 1 , kMAX = 2 , kMIN = 3 ,

kSUB = 4 , kDIV = 5 , kPOW = 6 , kFLOOR DIV = 7 ,

kAND = 8 , kOR = 9 , kXOR = 10 , kEQUAL = 11 ,

kGREATER = 12 , kLESS = 13 }

Enumerates the binary operations that may be performed by an ElementWise layer.

•

enum class GatherMode : int32 t { kDEFAULT = 0 , kELEMENT = 1 , kND = 2 }

Control form of IGatherLayer.

•

enum class RNNOperation : int32 t { kRELU = 0 , kTANH = 1 , kLSTM = 2 , kGRU = 3 }

Enumerates the RNN operations that may be performed by an RNN layer.

•

enum class RNNDirection : int32 t { kUNIDIRECTION = 0 , kBIDIRECTION = 1 }

Enumerates the RNN direction that may be performed by an RNN layer.

•

enum class RNNInputMode : int32 t { kLINEAR = 0 , kSKIP = 1 }

Enumerates the RNN input modes that may occur with an RNN layer.

•

enum class RNNGateType : int32 t {

kINPUT = 0 , kOUTPUT = 1 , kFORGET = 2 , kUPDATE = 3 ,

kRESET = 4 , kCELL = 5 , kHIDDEN = 6 }

Identifies an individual gate within an RNN cell.

•

enum class UnaryOperation : int32 t {

kEXP = 0 , kLOG = 1 , kSQRT = 2 , kRECIP = 3 ,

kABS = 4 , kNEG = 5 , kSIN = 6 , kCOS = 7 ,

kTAN = 8 , kSINH = 9 , kCOSH = 10 , kASIN = 11 ,

kACOS = 12 , kATAN = 13 , kASINH = 14 , kACOSH = 15 ,

kATANH = 16 , kCEIL = 17 , kFLOOR = 18 , kERF = 19 ,

kNOT = 20 , kSIGN = 21 , kROUND = 22 , kISINF = 23 }

Enumerates the unary operations that may be performed by a Unary layer.

•

enum class ReduceOperation : int32 t {

kSUM = 0 , kPROD = 1 , kMAX = 2 , kMIN = 3 ,

kAVG = 4 }

Enumerates the reduce operations that may be performed by a Reduce layer.

•

enum class SampleMode : int32 t {

kSTRICT BOUNDS = 0 , kDEFAULT = kSTRICT BOUNDS , kWRAP = 1 , kCLAMP = 2 ,

kFILL = 3 , kREFLECT = 4 }

Controls how ISliceLayer and IGridSample handle out-of-bounds coordinates.

•

enum class TopKOperation : int32 t { kMAX = 0 , kMIN = 1 }

Enumerates the operations that may be performed by a TopK layer.

•

enum class MatrixOperation : int32 t { kNONE , kTRANSPOSE , kVECTOR }

Enumerates the operations that may be performed on a tensor by IMatrixMultiplyLayer before multiplication.

•

enum class InterpolationMode : int32 t { kNEAREST = 0 , kLINEAR = 1 , kCUBIC = 2 }

Enumerates various modes of interpolation.

8.2 nvinfer1 Namespace Reference

•

enum class ResizeCoordinateTransformation : int32 t { kALIGN CORNERS = 0 , kASYMMETRIC = 1 ,

kHALF PIXEL = 2 }

The resize coordinate transformation function.

•

enum class ResizeSelector : int32 t { kFORMULA = 0 , kUPPER = 1 }

The coordinate selector when resize to single pixel output.

•

enum class ResizeRoundMode : int32 t { kHALF UP = 0 , kHALF DOWN = 1 , kFLOOR = 2 , kCEIL = 3 }

The rounding mode for nearest neighbor resize.

•

enum class LoopOutput : int32 t { kLAST VALUE = 0 , kCONCATENATE = 1 , kREVERSE = 2 }

Enum that describes kinds of loop outputs.

•

enum class TripLimit : int32 t { kCOUNT = 0 , kWHILE = 1 }

Enum that describes kinds of trip limits.

•

enum class FillOperation : int32 t { kLINSPACE = 0 , kRANDOM UNIFORM = 1 , kRANDOM NORMAL =

Enumerates the tensor fill operations that may performed by a fill layer.

•

enum class ScatterMode : int32 t { kELEMENT = 0 , kND = 1 }

Control form of IScatterLayer.

•

enum class BoundingBoxFormat : int32 t { kCORNER PAIRS = 0 , kCENTER SIZES = 1 }

Representation of bounding box data used for the Boxes input tensor in INMSLayer.

•

enum class CalibrationAlgoType : int32 t { kLEGACY CALIBRATION = 0 , kENTROPY CALIBRATION =

1 , kENTROPY CALIBRATION 2 = 2 , kMINMAX CALIBRATION = 3 }

Version of calibration algorithm to use.

•

enum class QuantizationFlag : int32 t { kCALIBRATE BEFORE FUSION = 0 }

List of valid flags for quantizing the network to int8.

•

enum class BuilderFlag : int32 t {

kFP16 = 0 , kINT8 = 1 , kDEBUG = 2 , kGPU FALLBACK = 3 ,

kSTRICT TYPES = 4 , kREFIT = 5 , kDISABLE TIMING CACHE = 6 , kTF32 = 7 ,

kSPARSE WEIGHTS = 8 , kSAFETY SCOPE = 9 , kOBEY PRECISION CONSTRAINTS = 10 ,

kPREFER PRECISION CONSTRAINTS = 11 ,

kDIRECT IO = 12 , kREJECT EMPTY ALGORITHMS = 13 , kENABLE TACTIC HEURISTIC = 14 ,

kVERSION COMPATIBLE = 15 ,

kEXCLUDE LEAN RUNTIME = 16 , kFP8 = 17 }

List of valid modes that the builder can enable when creating an engine from a network definition.

•

enum class MemoryPoolType : int32 t {

kWORKSPACE = 0 , kDLA MANAGED SRAM = 1 , kDLA LOCAL DRAM = 2 , kDLA GLOBAL DRAM

=3,

kTACTIC DRAM = 4 }

The type for memory pools used by TensorRT.

•

enum class PreviewFeature : int32 t { kFASTER DYNAMIC SHAPES 0805 = 0 , kDISABLE EXTERNAL TACTIC SOURCES

= 1 , kPROFILE SHARING 0806 = 2 }

Define preview features.

•

enum class HardwareCompatibilityLevel : int32 t { kNONE = 0 , kAMPERE PLUS = 1 }

•

enum class NetworkDefinitionCreationFlag : int32 t { kEXPLICIT BATCH = 0 , kEXPLICIT PRECISION = 1

}

List of immutable network properties expressed at network creation time. NetworkDefinitionCreationFlag is used with

createNetworkV2() to specify immutable properties of the network. Creating a network without NetworkDefinition←↩

CreationFlag::kEXPLICIT BATCH flag has been deprecated.

•

enum class EngineCapability : int32 t {

kSTANDARD = 0 , kDEFAULT = kSTANDARD , kSAFETY = 1 , kSAFE GPU = kSAFETY ,

kDLA STANDALONE = 2 , kSAFE DLA = kDLA STANDALONE }

List of supported engine capability flows.

Namespace Documentation

•

enum class DimensionOperation : int32 t {

kSUM = 0 , kPROD = 1 , kMAX = 2 , kMIN = 3 ,

kSUB = 4 , kEQUAL = 5 , kLESS = 6 , kFLOOR DIV = 7 ,

kCEIL DIV = 8 }

An operation on two IDimensionExpr, which represent integer expressions used in dimension computations.

•

enum class TensorLocation : int32 t { kDEVICE = 0 , kHOST = 1 }

The location for tensor data storage, device or host.

•

enum class WeightsRole : int32 t {

kKERNEL = 0 , kBIAS = 1 , kSHIFT = 2 , kSCALE = 3 ,

kCONSTANT = 4 , kANY = 5 }

How a layer uses particular Weights.

•

enum class DeviceType : int32 t { kGPU , kDLA }

The device that this layer/network will execute on.

•

enum class TempfileControlFlag : int32 t { kALLOW IN MEMORY FILES = 0 , kALLOW TEMPORARY FILES

=1}

Flags used to control TensorRT's behavior when creating executable temporary files.

•

enum class OptProfileSelector : int32 t { kMIN = 0 , kOPT = 1 , kMAX = 2 }

When setting or querying optimization profile parameters (such as shape tensor inputs or dynamic dimensions), select

whether we are interested in the minimum, optimum, or maximum values for these parameters. The minimum and max-

imum specify the permitted range that is supported at runtime, while the optimum value is used for the kernel selection.

This should be the ”typical” value that is expected to occur at runtime.

•

enum class TacticSource : int32 t {

kCUBLAS = 0 , kCUBLAS LT = 1 , kCUDNN = 2 , kEDGE MASK CONVOLUTIONS = 3 ,

kJIT CONVOLUTIONS = 4 }

List of tactic sources for TensorRT.

•

enum class ProfilingVerbosity : int32 t {

kLAYER NAMES ONLY = 0 , kNONE = 1 , kDETAILED = 2 , kDEFAULT = kLAYER NAMES ONLY ,

kVERBOSE = kDETAILED }

List of verbosity levels of layer information exposed in NVTX annotations and in IEngineInspector.

•

enum class LayerInformationFormat : int32 t { kONELINE = 0 , kJSON = 1 }

The format in which the IEngineInspector prints the layer information.

•

enum class DataType : int32 t {

kFLOAT = 0 , kHALF = 1 , kINT8 = 2 , kINT32 = 3 ,

kBOOL = 4 , kUINT8 = 5 , kFP8 = 6 }

The type of weights and tensors.

•

enum class TensorFormat : int32 t {

kLINEAR = 0 , kCHW2 = 1 , kHWC8 = 2 , kCHW4 = 3 ,

kCHW16 = 4 , kCHW32 = 5 , kDHWC8 = 6 , kCDHW32 = 7 ,

kHWC = 8 , kDLA LINEAR = 9 , kDLA HWC4 = 10 , kHWC16 = 11 ,

kDHWC = 12 }

Format of the input/output tensors.

•

enum class AllocatorFlag : int32 t { kRESIZABLE = 0 }

•

enum class ErrorCode : int32 t {

kSUCCESS = 0 , kUNSPECIFIED ERROR = 1 , kINTERNAL ERROR = 2 , kINVALID ARGUMENT = 3 ,

kINVALID CONFIG = 4 , kFAILED ALLOCATION = 5 , kFAILED INITIALIZATION = 6 , kFAILED EXECUTION

=7,

kFAILED COMPUTATION = 8 , kINVALID STATE = 9 , kUNSUPPORTED STATE = 10 }

Error codes that can be returned by TensorRT during execution.

•

enum class TensorIOMode : int32 t { kNONE = 0 , kINPUT = 1 , kOUTPUT = 2 }

Definition of tensor IO Mode.

8.2 nvinfer1 Namespace Reference

• enum class PluginVersion : uint8 t { kV2 = 0 , kV2 EXT = 1 , kV2 IOEXT = 2 , kV2 DYNAMICEXT = 3 }

• enum class PluginFieldType : int32 t {

kFLOAT16 = 0 , kFLOAT32 = 1 , kFLOAT64 = 2 , kINT8 = 3 ,

kINT16 = 4 , kINT32 = 5 , kCHAR = 6 , kDIMS = 7 ,

kUNKNOWN = 8 }

The possible field types for custom layer.

Functions

•

template<> constexpr int32 t EnumMax< LayerType > () noexcept

•

template<> constexpr int32 t EnumMax< ScaleMode > () noexcept

•

template<> constexpr int32 t EnumMax< GatherMode > () noexcept

•

template<> constexpr int32 t EnumMax< RNNOperation > () noexcept

•

template<> constexpr int32 t EnumMax< RNNDirection > () noexcept

•

template<> constexpr int32 t EnumMax< RNNInputMode > () noexcept

•

template<> constexpr int32 t EnumMax< RNNGateType > () noexcept

•

template<> constexpr int32 t EnumMax< UnaryOperation > () noexcept

•

template<> constexpr int32 t EnumMax< ReduceOperation > () noexcept

•

template<> constexpr int32 t EnumMax< SampleMode > () noexcept

•

template<> constexpr int32 t EnumMax< TopKOperation > () noexcept

•

template<> constexpr int32 t EnumMax< MatrixOperation > () noexcept

•

template<> constexpr int32 t EnumMax< LoopOutput > () noexcept

•

template<> constexpr int32 t EnumMax< TripLimit > () noexcept

•

template<> constexpr int32 t EnumMax< FillOperation > () noexcept

•

template<> constexpr int32 t EnumMax< ScatterMode > () noexcept

•

template<> constexpr int32 t EnumMax< BoundingBoxFormat > () noexcept

•

template<> constexpr int32 t EnumMax< CalibrationAlgoType > () noexcept

•

template<> constexpr int32 t EnumMax< QuantizationFlag > () noexcept

•

template<> constexpr int32 t EnumMax< BuilderFlag > () noexcept

•

template<> constexpr int32 t EnumMax< MemoryPoolType > () noexcept

•

template<> constexpr int32 t EnumMax< NetworkDefinitionCreationFlag > () noexcept

•

nvinfer1::IPluginRegistry ∗ getBuilderPluginRegistry (nvinfer1::EngineCapability capability) noexcept

Return the plugin registry for building a Standard engine, or nullptr if no registry exists.

•

nvinfer1::safe::IPluginRegistry ∗ getBuilderSafePluginRegistry (nvinfer1::EngineCapability capability) noex-

cept

Return the plugin registry for building a Safety engine, or nullptr if no registry exists.

•

template<> constexpr int32 t EnumMax< DimensionOperation > () noexcept

Maximum number of elements in DimensionOperation enum.

•

template<> constexpr int32 t EnumMax< WeightsRole > () noexcept

Maximum number of elements in WeightsRole enum.

•

template<> constexpr int32 t EnumMax< DeviceType > () noexcept

Maximum number of elements in DeviceType enum.

•

template<> constexpr int32 t EnumMax< TempfileControlFlag > () noexcept

Maximum number of elements in TempfileControlFlag enum.

•

template<> constexpr int32 t EnumMax< OptProfileSelector > () noexcept

Number of different values of OptProfileSelector enum.

•

template<> constexpr int32 t EnumMax< TacticSource > () noexcept

Maximum number of tactic sources in TacticSource enum.

•

template<> constexpr int32 t EnumMax< ProfilingVerbosity > () noexcept

Namespace Documentation

Maximum number of profile verbosity levels in ProfilingVerbosity enum.

• template<> constexpr int32 t EnumMax< LayerInformationFormat > () noexcept

• template<typename T >

constexpr int32 t EnumMax () noexcept

Maximum number of elements in an enumeration type.

8.2.1

Detailed Description

The TensorRT API version 1 namespace.

8.2.2

Typedef Documentation

8.2.2.1

AllocatorFlags

using nvinfer1::AllocatorFlags = typedef uint32 t

8.2.2.2

AsciiChar

using nvinfer1::AsciiChar = typedef char t

AsciiChar is the type used by TensorRT to represent valid ASCII characters. This type is used by IPluginV2,

PluginField, IPluginCreator, IPluginRegistry, and ILogger due to their use in automotive safety context.

8.2.2.3

BuilderFlags

using nvinfer1::BuilderFlags = typedef uint32 t

Represents one or more BuilderFlag values using binary OR operations, e.g., 1U << BuilderFlag::kFP16 | 1U <<

BuilderFlag::kDEBUG.

See also

IBuilderConfig::setFlags(), IBuilderConfig::getFlags()

8.2.2.4

char t

using nvinfer1::char t = typedef char

char t is the type used by TensorRT to represent all valid characters.

8.2.2.5

Dims

using nvinfer1::Dims = typedef Dims32

Alias for Dims32.

8.2 nvinfer1 Namespace Reference

Warning

: This alias might change in the future.

8.2.2.6

NetworkDefinitionCreationFlags

using nvinfer1::NetworkDefinitionCreationFlags = typedef uint32 t

Represents one or more NetworkDefinitionCreationFlag flags using binary OR operations.

e.g., 1U <<

NetworkDefinitionCreationFlag::kEXPLICIT BATCH.

See also

IBuilder::createNetworkV2

8.2.2.7

PluginFormat

using nvinfer1::PluginFormat = typedef TensorFormat

PluginFormat is reserved for backward compatibility.

See also

IPluginV2::supportsFormat()

8.2.2.8

QuantizationFlags

using nvinfer1::QuantizationFlags = typedef uint32 t

Represents one or more QuantizationFlag values using binary OR operations.

See also

IBuilderConfig::getQuantizationFlags(), IBuilderConfig::setQuantizationFlags()

Namespace Documentation

8.2.2.9

ResizeMode

using nvinfer1::ResizeMode = typedef InterpolationMode

Deprecated Deprecated in TensorRT 8.5. Superseded by InterpolationMode.

8.2.2.10

SliceMode

using nvinfer1::SliceMode = typedef SampleMode

Deprecated Deprecated in TensorRT 8.5. Superseded by SampleMode.

8.2.2.11

TacticSources

using nvinfer1::TacticSources = typedef uint32 t

Represents a collection of one or more TacticSource values combine using bitwise-OR operations.

See also

IBuilderConfig::setTacticSources(), IBuilderConfig::getTacticSources()

8.2.2.12

TempfileControlFlags

using nvinfer1::TempfileControlFlags = typedef uint32 t

Represents a collection of one or more TempfileControlFlag values combined using bitwise-OR operations.

See also

TempfileControlFlag, IRuntime::setTempfileControlFlags(), IRuntime::getTempfileControlFlags()

8.2 nvinfer1 Namespace Reference

8.2.2.13

TensorFormats

using nvinfer1::TensorFormats = typedef uint32 t

It is capable of representing one or more TensorFormat by binary OR operations, e.g., 1U << TensorFormat::kCHW4

| 1U << TensorFormat::kCHW32.

See also

ITensor::getAllowedFormats(), ITensor::setAllowedFormats(),

8.2.3

Enumeration Type Documentation

8.2.3.1

ActivationType

enum class nvinfer1::ActivationType : int32 t [strong]

Enumerates the types of activation to perform in an activation layer.

Enumerator

kRELU

Rectified linear activation.

kSIGMOID

Sigmoid activation.

kTANH

TanH activation.

kLEAKY RELU

LeakyRelu activation: x>=0 ? x : alpha ∗ x.

kELU

Elu activation: x>=0 ? x : alpha ∗ (exp(x) - 1).

kSELU

Selu activation: x>0 ? beta ∗ x : beta ∗ (alpha∗exp(x) - alpha)

kSOFTSIGN

Softsign activation: x / (1+|x|)

kSOFTPLUS

Parametric softplus activation: alpha∗log(exp(beta∗x)+1)

kCLIP

Clip activation: max(alpha, min(beta, x))

kHARD SIGMOID

Hard sigmoid activation: max(0, min(1, alpha∗x+beta))

kSCALED TANH

Scaled tanh activation: alpha∗tanh(beta∗x)

kTHRESHOLDED RELU

Thresholded ReLU activation: x>alpha ? x : 0.

8.2.3.2

AllocatorFlag

enum class nvinfer1::AllocatorFlag : int32 t [strong]

Namespace Documentation

Enumerator

kRESIZABLE

TensorRT may call realloc() on this allocation.

8.2.3.3

BoundingBoxFormat

enum class nvinfer1::BoundingBoxFormat : int32 t [strong]

Representation of bounding box data used for the Boxes input tensor in INMSLayer.

See also

INMSLayer

Enumerator

kCORNER PAIRS

(x1, y1, x2, y2) where (x1, y1) and (x2, y2) are any pair of diagonal corners

kCENTER SIZES

(x center, y center, width, height) where (x center, y center) is the center point of the box

8.2.3.4

BuilderFlag

enum class nvinfer1::BuilderFlag : int32 t [strong]

List of valid modes that the builder can enable when creating an engine from a network definition.

See also

IBuilderConfig::setFlags(), IBuilderConfig::getFlags()

Enumerator

kFP16

Enable FP16 layer selection, with FP32 fallback.

kINT8

Enable Int8 layer selection, with FP32 fallback with FP16 fallback if

kFP16 also specified.

kDEBUG

Enable debugging of layers via synchronizing after every layer.

kGPU FALLBACK

Enable layers marked to execute on GPU if layer cannot execute on

DLA.

8.2 nvinfer1 Namespace Reference

Enumerator

kSTRICT TYPES

Legacy flag with effect similar to setting all of these three flags:

* kPREFER_PRECISION_CONSTRAINTS

* kDIRECT_IO

* kREJECT_EMPTY_ALGORITHMS

except that if the direct I/O requirement cannot be met and

kDIRECT_IO was

instead of the build failing, the build falls back as if kD

IRECT_IO was no

\deprecated Deprecated in TensorRT 8.2.

kREFIT

Enable building a refittable engine.

kDISABLE TIMING CACHE

Disable reuse of timing information across identical layers.

kTF32

Allow (but not require) computations on tensors of type

DataType::kFLOAT to use TF32. TF32 computes inner products by

rounding the inputs to 10-bit mantissas before multiplying, but

accumulates the sum using 23-bit mantissas. Enabled by default.

kSPARSE WEIGHTS

Allow the builder to examine weights and use optimized functions

when weights have suitable sparsity.

kSAFETY SCOPE

Change the allowed parameters in the

EngineCapability::kSTANDARD flow to match the restrictions that

EngineCapability::kSAFETY check against for DeviceType::kGPU

and EngineCapability::kDLA STANDALONE check against the

DeviceType::kDLA case. This flag is forced to true if

EngineCapability::kSAFETY at build time if it is unset.

This flag is only supported in NVIDIA Drive(R) products.

kOBEY PRECISION CONSTRAINTS

Require that layers execute in specified precisions. Build fails

otherwise.

kPREFER PRECISION CONSTRAINTS

Prefer that layers execute in specified precisions. Fall back (with

warning) to another precision if build would otherwise fail.

kDIRECT IO

Require that no reformats be inserted between a layer and a network

I/O tensor for which ITensor::setAllowedFormats was called. Build

fails if a reformat is required for functional correctness.

kREJECT EMPTY ALGORITHMS

Fail if IAlgorithmSelector::selectAlgorithms returns an empty set of

algorithms.

kENABLE TACTIC HEURISTIC

Enable heuristic-based tactic selection for shorter engine generation

time. The engine may not be as performant as when built with a

profiling-based builder.

This flag is only supported by NVIDIA Ampere and later GPUs.

Deprecated Superseded by builder optimization level 2.

Deprecated in TensorRT 8.6

kVERSION COMPATIBLE

Restrict to lean runtime operators to provide version forward

compatibility for the plan.

Using this flag with ICudaEngine::serialize() and

BuilderFlag::kREFIT would result in error. This flag is only

supported by NVIDIA Volta and later GPUs. This flag is not

supported in NVIDIA Drive(R) products. This flag is not supported

with implicit batch mode. Network must be created with

NetworkDefinitionCreationFlag::kEXPLICIT BATCH.

Namespace Documentation

Enumerator

kEXCLUDE LEAN RUNTIME

Exclude lean runtime from the plan when version forward

compatability is enabled. By default, this flag is unset, so the lean

runtime will be included in the plan.

If BuilderFlag::kVERSION COMPATIBLE is not set then the value

of this flag will be ignored.

This flag is not supported with implicit batch mode. Network must

be created with

NetworkDefinitionCreationFlag::kEXPLICIT BATCH.

kFP8

Enable FP8 layer selection, with FP32 fallback.

Warning

kFP8 is not supported yet and will result in an error or

undefined behavior.

8.2.3.5

CalibrationAlgoType

enum class nvinfer1::CalibrationAlgoType : int32 t [strong]

Version of calibration algorithm to use.

enum CalibrationAlgoType

Enumerator

kLEGACY CALIBRATION

kENTROPY CALIBRATION

kENTROPY ←↩

CALIBRATION 2

kMINMAX CALIBRATION

8.2.3.6

DataType

enum class nvinfer1::DataType : int32 t [strong]

The type of weights and tensors.

Enumerator

kFLOAT

32-bit floating point format.

kHALF

IEEE 16-bit floating-point format.

8.2 nvinfer1 Namespace Reference

Enumerator

kINT8

Signed 8-bit integer representing a quantized floating-point value.

kINT32

Signed 32-bit integer format.

kBOOL

8-bit boolean. 0 = false, 1 = true, other values undefined.

kUINT8

Unsigned 8-bit integer format. Cannot be used to represent quantized floating-point values. Use the

IdentityLayer to convert kUINT8 network-level inputs to {kFLOAT, kHALF} prior to use with other

TensorRT layers, or to convert intermediate output before kUINT8 network-level outputs from

{kFLOAT, kHALF} to kUINT8. kUINT8 conversions are only supported for {kFLOAT, kHALF}.

kUINT8 to {kFLOAT, kHALF} conversion will convert the integer values to equivalent floating point

values. {kFLOAT, kHALF} to kUINT8 conversion will convert the floating point values to integer

values by truncating towards zero. This conversion has undefined behavior for floating point values

outside the range [0.0f, 256.0f) after truncation. kUINT8 conversions are not supported for {kINT8,

kINT32, kBOOL}.

kFP8

Signed 8-bit floating point with 1 sign bit, 4 exponent bits, 3 mantissa bits, and exponent-bias 7.

Warning

kFP8 is not supported yet and will result in an error or undefined behavior.

8.2.3.7

DeviceType

enum class nvinfer1::DeviceType : int32 t [strong]

The device that this layer/network will execute on.

Enumerator

kGPU

GPU Device.

kDLA

DLA Core.

8.2.3.8

DimensionOperation

enum class nvinfer1::DimensionOperation : int32 t [strong]

An operation on two IDimensionExpr, which represent integer expressions used in dimension computations.

For example, given two IDimensionExpr x and y and an IExprBuilder& eb, eb.operation(DimensionOperation::kSUM,

x, y) creates a representation of x+y.

See also

IDimensionExpr, IExprBuilder

Namespace Documentation

Enumerator

kSUM

Sum of the two operands.

kPROD

Product of the two operands.

kMAX

Maximum of the two operands.

kMIN

Minimum of the two operands.

kSUB

Substract the second element from the first.

kEQUAL

1 if operands are equal, 0 otherwise.

kLESS

1 if first operand is less than second operand, 0 otherwise.

kFLOOR DIV

Floor division of the first element by the second.

kCEIL DIV

Division rounding up.

8.2.3.9

ElementWiseOperation

enum class nvinfer1::ElementWiseOperation : int32 t [strong]

Enumerates the binary operations that may be performed by an ElementWise layer.

Operations kAND, kOR, and kXOR must have inputs of DataType::kBOOL.

Operation kPOW must have inputs of DataType::kFLOAT, DataType::kHALF, or DataType::kINT8.

All other operations must have inputs of DataType::kFLOAT, DataType::kHALF, DataType::kINT8, or

DataType::kINT32.

See also

IElementWiseLayer

Enumerator

kSUM

Sum of the two elements.

kPROD

Product of the two elements.

kMAX

Maximum of the two elements.

kMIN

Minimum of the two elements.

kSUB

Subtract the second element from the first.

kDIV

Divide the first element by the second.

kPOW

The first element to the power of the second element.

kFLOOR DIV

Floor division of the first element by the second.

kAND

Logical AND of two elements.

kOR

Logical OR of two elements.

kXOR

Logical XOR of two elements.

kEQUAL

Check if two elements are equal.

kGREATER

Check if element in first tensor is greater than corresponding element in second tensor.

kLESS

Check if element in first tensor is less than corresponding element in second tensor.

8.2 nvinfer1 Namespace Reference

8.2.3.10

EngineCapability

enum class nvinfer1::EngineCapability : int32 t [strong]

List of supported engine capability flows.

The EngineCapability determines the restrictions of a network during build time and what runtime it targets. When

BuilderFlag::kSAFETY SCOPE is not set (by default), EngineCapability::kSTANDARD does not provide any restric-

tions on functionality and the resulting serialized engine can be executed with TensorRT's standard runtime APIs in the

nvinfer1 namespace. EngineCapability::kSAFETY provides a restricted subset of network operations that are safety

certified and the resulting serialized engine can be executed with TensorRT's safe runtime APIs in the nvinfer1::safe

namespace. EngineCapability::kDLA STANDALONE provides a restricted subset of network operations that are DLA

compatible and the resulting serialized engine can be executed using standalone DLA runtime APIs. See sampleCudla

for an example of integrating cuDLA APIs with TensorRT APIs.

Enumerator

kSTANDARD

Standard: TensorRT flow without targeting the safety runtime. This flow supports both

DeviceType::kGPU and DeviceType::kDLA.

kDEFAULT

Deprecated Deprecated in TensorRT 8.0. Superseded by kSTANDARD.

kSAFETY

Safety: TensorRT flow with restrictions targeting the safety runtime. See safety

documentation for list of supported layers and formats. This flow supports only

DeviceType::kGPU.

This flag is only supported in NVIDIA Drive(R) products.

kSAFE GPU

Deprecated Deprecated in TensorRT 8.0. Superseded by kSAFETY.

kDLA STANDALONE

DLA Standalone: TensorRT flow with restrictions targeting external, to TensorRT,

DLA runtimes. See DLA documentation for list of supported layers and formats. This

flow supports only DeviceType::kDLA.

kSAFE DLA

Deprecated Deprecated in TensorRT 8.0. Superseded by kDLA STANDALONE.

8.2.3.11

ErrorCode

enum class nvinfer1::ErrorCode : int32 t [strong]

Error codes that can be returned by TensorRT during execution.

Namespace Documentation

Enumerator

kSUCCESS

Execution completed successfully.

kUNSPECIFIED ERROR

An error that does not fall into any other category. This error is included for

forward compatibility.

kINTERNAL ERROR

A non-recoverable TensorRT error occurred. TensorRT is in an invalid internal

state when this error is emitted and any further calls to TensorRT will result in

undefined behavior.

kINVALID ARGUMENT

An argument passed to the function is invalid in isolation. This is a violation of

the API contract.

kINVALID CONFIG

An error occurred when comparing the state of an argument relative to other

arguments. For example, the dimensions for concat differ between two tensors

outside of the channel dimension. This error is triggered when an argument is

correct in isolation, but not relative to other arguments. This is to help to

distinguish from the simple errors from the more complex errors. This is a

violation of the API contract.

kFAILED ALLOCATION

An error occurred when performing an allocation of memory on the host or the

device. A memory allocation error is normally fatal, but in the case where the

application provided its own memory allocation routine, it is possible to increase

the pool of available memory and resume execution.

kFAILED INITIALIZATION

One, or more, of the components that TensorRT relies on did not initialize

correctly. This is a system setup issue.

kFAILED EXECUTION

An error occurred during execution that caused TensorRT to end prematurely,

either an asynchronous error or other execution errors reported by CUDA/DLA.

In a dynamic system, the data can be thrown away and the next frame can be

processed or execution can be retried. This is either an execution error or a

memory error.

kFAILED COMPUTATION

An error occurred during execution that caused the data to become corrupted, but

execution finished. Examples of this error are NaN squashing or integer

overflow. In a dynamic system, the data can be thrown away and the next frame

can be processed or execution can be retried. This is either a data corruption

error, an input error, or a range error. This is not used in safety but may be used

in standard.

kINVALID STATE

TensorRT was put into a bad state by incorrect sequence of function calls. An

example of an invalid state is specifying a layer to be DLA only without GPU

fallback, and that layer is not supported by DLA. This can occur in situations

where a service is optimistically executing networks for multiple different

configurations without checking proper error configurations, and instead

throwing away bad configurations caught by TensorRT. This is a violation of the

API contract, but can be recoverable.

Example of a recovery: GPU fallback is disabled and conv layer with large

filter(63x63) is specified to run on DLA. This will fail due to DLA not

supporting the large kernel size. This can be recovered by either turning on GPU

fallback or setting the layer to run on the GPU.

8.2 nvinfer1 Namespace Reference

Enumerator

kUNSUPPORTED STATE

An error occurred due to the network not being supported on the device due to

constraints of the hardware or system. An example is running a unsafe layer in a

safety certified context, or a resource requirement for the current network is

greater than the capabilities of the target device. The network is otherwise

correct, but the network and hardware combination is problematic. This can be

recoverable. Examples:

• Scratch space requests larger than available device memory and can be

recovered by increasing allowed workspace size.

• Tensor size exceeds the maximum element count and can be recovered by

reducing the maximum batch size.

8.2.3.12

FillOperation

enum class nvinfer1::FillOperation : int32 t [strong]

Enumerates the tensor fill operations that may performed by a fill layer.

See also

IFillLayer

Enumerator

kLINSPACE

Generate evenly spaced numbers over a specified interval.

kRANDOM UNIFORM

Generate a tensor with random values drawn from a uniform distribution.

kRANDOM NORMAL

Generate a tensor with random values drawn from a normal distribution.

8.2.3.13

GatherMode

enum class nvinfer1::GatherMode : int32 t [strong]

Control form of IGatherLayer.

See also

IGatherLayer

Namespace Documentation

Enumerator

kDEFAULT

Similar to ONNX Gather.

kELEMENT

Similar to ONNX GatherElements.

kND

Similar to ONNX GatherND.

8.2.3.14

HardwareCompatibilityLevel

enum class nvinfer1::HardwareCompatibilityLevel : int32 t [strong]

Describes requirements of compatibility with GPU architectures other than that of the GPU on which the engine was

built. Levels except kNONE are only supported for engines built on NVIDIA Ampere and later GPUs. Note that

compatibility with future hardware depends on CUDA forward compatibility support.

Enumerator

kNONE

Do not require hardware compatibility with GPU architectures other than that of the GPU on

which the engine was built.

kAMPERE PLUS

Require that the engine is compatible with Ampere and newer GPUs. This will limit the max

shared memory usage to 48KiB, may reduce the number of available tactics for each layer,

and may prevent some fusions from occurring. Thus this can decrease the performance,

especially for tf32 models. This option will disable cuDNN, cuBLAS, and cuBLAS LT as

tactic sources.

8.2.3.15

InterpolationMode

enum class nvinfer1::InterpolationMode : int32 t [strong]

Enumerates various modes of interpolation.

Enumerator

kNEAREST

ND (0 < N <= 8) nearest neighbor resizing.

kLINEAR

Supports linear (1D), bilinear (2D), and trilinear (3D) interpolation.

kCUBIC

Supports bicubic (2D) interpolation.

8.2.3.16

LayerInformationFormat

enum class nvinfer1::LayerInformationFormat : int32 t [strong]

8.2 nvinfer1 Namespace Reference

The format in which the IEngineInspector prints the layer information.

See also

IEngineInspector::getLayerInformation(), IEngineInspector::getEngineInformation()

Enumerator

kONELINE

Print layer information in one line per layer.

kJSON

Print layer information in JSON format.

8.2.3.17

LayerType

enum class nvinfer1::LayerType : int32 t [strong]

The type values of layer classes.

See also

ILayer::getType()

Enumerator

kCONVOLUTION

Convolution layer.

kFULLY CONNECTED

Fully connected layer.

kACTIVATION

Activation layer.

kPOOLING

Pooling layer.

kLRN

LRN layer.

kSCALE

Scale layer.

kSOFTMAX

SoftMax layer.

kDECONVOLUTION

Deconvolution layer.

kCONCATENATION

Concatenation layer.

kELEMENTWISE

Elementwise layer.

kPLUGIN

Plugin layer.

kUNARY

UnaryOp operation Layer.

kPADDING

Padding layer.

kSHUFFLE

Shuffle layer.

kREDUCE

Reduce layer.

kTOPK

TopK layer.

kGATHER

Gather layer.

kMATRIX MULTIPLY

Matrix multiply layer.

kRAGGED SOFTMAX

Ragged softmax layer.

Namespace Documentation

Enumerator

kCONSTANT

Constant layer.

kRNN V2

RNNv2 layer.

kIDENTITY

Identity layer.

kPLUGIN V2

PluginV2 layer.

kSLICE

Slice layer.

kSHAPE

Shape layer.

kPARAMETRIC RELU

Parametric ReLU layer.

kRESIZE

Resize Layer.

kTRIP LIMIT

Loop Trip limit layer.

kRECURRENCE

Loop Recurrence layer.

kITERATOR

Loop Iterator layer.

kLOOP OUTPUT

Loop output layer.

kSELECT

Select layer.

kFILL

Fill layer.

kQUANTIZE

Quantize layer.

kDEQUANTIZE

Dequantize layer.

kCONDITION

Condition layer.

kCONDITIONAL INPUT

Conditional Input layer.

kCONDITIONAL OUTPUT

Conditional Output layer.

kSCATTER

Scatter layer.

kEINSUM

Einsum layer.

kASSERTION

Assertion layer.

kONE HOT

OneHot layer.

kNON ZERO

NonZero layer.

kGRID SAMPLE

Grid sample layer.

kNMS

NMS layer.

kREVERSE SEQUENCE

Reverse sequence layer.

kNORMALIZATION

Normalization layer.

kCAST

Cast layer.

8.2.3.18

LoopOutput

enum class nvinfer1::LoopOutput : int32 t [strong]

Enum that describes kinds of loop outputs.

Enumerator

kLAST VALUE

Output value is value of tensor for last iteration.

kCONCATENATE

Output value is concatenation of values of tensor for each iteration, in forward order.

kREVERSE

Output value is concatenation of values of tensor for each iteration, in reverse order.

8.2 nvinfer1 Namespace Reference

8.2.3.19

MatrixOperation

enum class nvinfer1::MatrixOperation : int32 t [strong]

Enumerates the operations that may be performed on a tensor by IMatrixMultiplyLayer before multiplication.

Enumerator

kNONE

Treat x as a matrix if it has two dimensions, or as a collection of matrices if x has more than

two dimensions, where the last two dimensions are the matrix dimensions. x must have at least

two dimensions.

kTRANSPOSE

Like kNONE, but transpose the matrix dimensions.

kVECTOR

Treat x as a vector if it has one dimension, or as a collection of vectors if x has more than one

dimension. x must have at least one dimension.

The first input tensor with dimensions [M,K] used with MatrixOperation::kVECTOR is

equivalent to a tensor with dimensions [M, 1, K] with MatrixOperation::kNONE, i.e. is treated

as M row vectors of length K, or dimensions [M, K, 1] with MatrixOperation::kTRANSPOSE.

The second input tensor with dimensions [M,K] used with MatrixOperation::kVECTOR is

equivalent to a tensor with dimensions [M, K, 1] with MatrixOperation::kNONE, i.e. is treated

as M column vectors of length K, or dimensions [M, 1, K] with

MatrixOperation::kTRANSPOSE.

8.2.3.20

MemoryPoolType

enum class nvinfer1::MemoryPoolType : int32 t [strong]

The type for memory pools used by TensorRT.

See also

IBuilderConfig::setMemoryPoolLimit, IBuilderConfig::getMemoryPoolLimit

Enumerator

kWORKSPACE

kWORKSPACE is used by TensorRT to store intermediate buffers within an

operation. This is equivalent to the deprecated

IBuilderConfig::setMaxWorkspaceSize and overrides that value. This defaults to

max device memory. Set to a smaller value to restrict tactics that use over the

threshold en masse. For more targeted removal of tactics use the

IAlgorithmSelector interface.

kDLA MANAGED SRAM

kDLA MANAGED SRAM is a fast software managed RAM used by DLA to

communicate within a layer. The size of this pool must be at least 4 KiB and must

be a power of 2. This defaults to 1 MiB. Orin has capacity of 1 MiB per core, and

Xavier shares 4 MiB across all of its accelerator cores.

Namespace Documentation

Enumerator

kDLA LOCAL DRAM

kDLA LOCAL DRAM is host RAM used by DLA to share intermediate tensor

data across operations. The size of this pool must be at least 4 KiB and must be a

power of 2. This defaults to 1 GiB.

kDLA GLOBAL DRAM

kDLA GLOBAL DRAM is host RAM used by DLA to store weights and

metadata for execution. The size of this pool must be at least 4 KiB and must be a

power of 2. This defaults to 512 MiB.

kTACTIC DRAM

kTACTIC DRAM is the device DRAM used by the optimizer to run tactics. On

embedded devices, where host and device memory are unified, this includes all

host memory required by TensorRT to build the network up to the point of each

memory allocation. This defaults to 75% of totalGlobalMem as reported by

cudaGetDeviceProperties when cudaGetDeviceProperties.embedded is true, and

100% otherwise.

8.2.3.21

NetworkDefinitionCreationFlag

enum class nvinfer1::NetworkDefinitionCreationFlag : int32 t [strong]

List of immutable network properties expressed at network creation time. NetworkDefinitionCreationFlag is

used with createNetworkV2() to specify immutable properties of the network. Creating a network without

NetworkDefinitionCreationFlag::kEXPLICIT BATCH flag has been deprecated.

See also

IBuilder::createNetworkV2

Enumerator

kEXPLICIT BATCH

Mark the network to be an explicit batch network. Dynamic shape support requires

that the kEXPLICIT BATCH flag is set. With dynamic shapes, any of the input

dimensions can vary at run-time, and there are no implicit dimensions in the network

specification. Varying dimensions are specified by using the wildcard dimension

value -1.

kEXPLICIT PRECISION

Deprecated. This flag has no effect now, but is only kept for backward compatability.

8.2.3.22

OptProfileSelector

enum class nvinfer1::OptProfileSelector : int32 t [strong]

When setting or querying optimization profile parameters (such as shape tensor inputs or dynamic dimensions), select

whether we are interested in the minimum, optimum, or maximum values for these parameters. The minimum and

maximum specify the permitted range that is supported at runtime, while the optimum value is used for the kernel

selection. This should be the ”typical” value that is expected to occur at runtime.

8.2 nvinfer1 Namespace Reference

See also

IOptimizationProfile::setDimensions(), IOptimizationProfile::setShapeValues()

Enumerator

kMIN

This is used to set or get the minimum permitted value for dynamic dimensions etc.

kOPT

This is used to set or get the value that is used in the optimization (kernel selection).

kMAX

This is used to set or get the maximum permitted value for dynamic dimensions etc.

8.2.3.23

PaddingMode

enum class nvinfer1::PaddingMode : int32 t [strong]

Enumerates the modes of padding to perform in convolution, deconvolution and pooling layer, padding mode takes

precedence if setPaddingMode() and setPrePadding() are also used.

There are three padding styles, EXPLICIT, SAME, and CAFFE, with each style having two variants. The EXPLICIT

and CAFFE styles determine if the final sampling location is used or not. The SAME style determine if the asymmetry

in the padding is on the pre or post padding.

Shorthand:

I = dimensions of input image.

B = prePadding, before the image data. For deconvolution, prePadding is set before output.

A = postPadding, after the image data. For deconvolution, postPadding is set after output.

P = delta between input and output

S = stride

F = filter

O = output

D = dilation

M = I + B + A ; The image data plus any padding

DK = 1 + D * (F - 1)

Formulas for Convolution:

• EXPLICIT ROUND DOWN:

O = floor((M - DK) / S) + 1

• CAFFE ROUND DOWN:

O = floor((I + B * 2 - DK) / S) + 1

• EXPLICIT ROUND UP:

O = ceil((M - DK) / S) + 1

• CAFFE ROUND UP:

O = ceil((I + B * 2 - DK) / S) + 1

• SAME UPPER:

O = ceil(I / S)

P = floor((I - 1) / S) * S + DK - I;

B = floor(P / 2)

A = P - B

• SAME LOWER:

O = ceil(I / S)

P = floor((I - 1) / S) * S + DK - I;

A = floor(P / 2)

B = P - A

Namespace Documentation

Formulas for Deconvolution:

• EXPLICIT ROUND DOWN:

• CAFFE ROUND DOWN:

• EXPLICIT ROUND UP:

• CAFFE ROUND UP:

O = (I - 1) * S + DK - (B + A)

• SAME UPPER:

O = min(I * S, (I - 1) * S + DK)

P = max(DK - S, 0)

B = floor(P / 2)

A = P - B

• SAME LOWER:

O = min(I * S, (I - 1) * S + DK)

P = max(DK - S, 0)

A = floor(P / 2)

B = P - A

Formulas for Pooling:

• EXPLICIT ROUND DOWN:

O = floor((M - F) / S) + 1

• EXPLICIT ROUND UP:

O = ceil((M - F) / S) + 1

• SAME UPPER:

O = ceil(I / S)

P = floor((I - 1) / S) * S + F - I;

B = floor(P / 2)

A = P - B

• SAME LOWER:

O = ceil(I / S)

P = floor((I - 1) / S) * S + F - I;

A = floor(P / 2)

B = P - A

• CAFFE ROUND DOWN:

EXPLICIT ROUND DOWN - ((EXPLICIT ROUND DOWN - 1) * S >= I + B)

• CAFFE ROUND UP:

EXPLICIT ROUND UP - ((EXPLICIT ROUND UP - 1) * S >= I + B)

Pooling Example 1:

Given I = {6, 6}, B = {3, 3}, A = {2, 2}, S = {2, 2}, F = {3, 3}. What is O?

(B, A can be calculated for SAME UPPER and SAME LOWER mode)

• EXPLICIT ROUND DOWN:

Computation:

M = {6, 6} + {3, 3} + {2, 2} ==> {11, 11}

O ==> floor((M - F) / S) + 1

==> floor(({11, 11} - {3, 3}) / {2, 2}) + {1, 1}

==> floor({8, 8} / {2, 2}) + {1, 1}

==> {5, 5}

• EXPLICIT ROUND UP:

Computation:

M = {6, 6} + {3, 3} + {2, 2} ==> {11, 11}

O ==> ceil((M - F) / S) + 1

==> ceil(({11, 11} - {3, 3}) / {2, 2}) + {1, 1}

==> ceil({8, 8} / {2, 2}) + {1, 1}

==> {5, 5}

The sample points are {0, 2, 4, 6, 8} in each dimension.

8.2 nvinfer1 Namespace Reference

• SAME UPPER:

Computation:

I = {6, 6}

S = {2, 2}

O = ceil(I / S) = {3, 3}

P = floor((I - 1) / S) * S + F - I

==> floor(({6, 6} - {1, 1}) / {2, 2}) * {2, 2} + {3, 3} - {6, 6}

==> {4, 4} + {3, 3} - {6, 6}

==> {1, 1}

B = floor({1, 1} / {2, 2})

==> {0, 0}

A = {1, 1} - {0, 0}

==> {1, 1}

• SAME LOWER:

Computation:

I = {6, 6}

S = {2, 2}

O = ceil(I / S) = {3, 3}

P = floor((I - 1) / S) * S + F - I

==> {1, 1}

A = floor({1, 1} / {2, 2})

==> {0, 0}

B = {1, 1} - {0, 0}

==> {1, 1}

The sample pointers are {0, 2, 4} in each dimension. SAMPLE UPPER has {O0, O1, O2, pad} in output in each

dimension. SAMPLE LOWER has {pad, O0, O1, O2} in output in each dimension.

Pooling Example 2:

Given I = {6, 6}, B = {3, 3}, A = {3, 3}, S = {2, 2}, F = {3, 3}. What is O?

• CAFFE ROUND DOWN:

Computation:

M = {6, 6} + {3, 3} + {3, 3} ==> {12, 12}

EXPLICIT ROUND DOWN ==> floor((M - F) / S) + 1

==> floor(({12, 12} - {3, 3}) / {2, 2}) + {1, 1}

==> {5, 5}

DIFF = (((EXPLICIT ROUND DOWN - 1) * S >= I + B) ? {1, 1} : {0, 0})

==> ({5, 5} - {1, 1}) * {2, 2} >= {6, 6} + {3, 3} ? {1, 1} : {0,0}

==> {0, 0}

O ==> EXPLICIT ROUND DOWN - DIFF

==> {5, 5} - {0, 0}

==> {5, 5}

• CAFFE ROUND UP:

Computation:

M = {6, 6} + {3, 3} + {3, 3} ==> {12, 12}

EXPLICIT ROUND UP ==> ceil((M - F) / S) + 1

==> ceil(({12, 12} - {3, 3}) / {2, 2}) + {1, 1}

==> {6, 6}

DIFF = (((EXPLICIT ROUND UP - 1) * S >= I + B) ? {1, 1} : {0, 0})

==> ({6, 6} - {1, 1}) * {2, 2} >= {6, 6} + {3, 3} ? {1, 1} : {0,0}

==> {1, 1}

O ==> EXPLICIT ROUND UP - DIFF

==> {6, 6} - {1, 1}

==> {5, 5}

The sample points are {0, 2, 4, 6, 8} in each dimension.

CAFFE ROUND DOWN and CAFFE ROUND UP have two restrictions each on usage with pooling operations. This

will cause getDimensions to return an empty dimension and also to reject the network at validation time.

For more information on original reference code, see https://github.com/BVLC/caffe/blob/master/src/caffe/lay

layer.cpp

• Restriction 1:

CAFFE ROUND DOWN: B >= F is an error if (B - S) < F

CAFFE ROUND UP: (B + S) >= (F + 1) is an error if B < (F + 1)

• Restriction 2:

CAFFE ROUND DOWN: (B - S) >= F is an error if B >= F

CAFFE ROUND UP: B >= (F + 1) is an error if (B + S) >= (F + 1)

Namespace Documentation

Enumerator

kEXPLICIT ROUND DOWN

Use explicit padding, rounding output size down.

kEXPLICIT ROUND UP

Use explicit padding, rounding output size up.

kSAME UPPER

Use SAME padding, with prePadding <= postPadding.

kSAME LOWER

Use SAME padding, with prePadding >= postPadding.

kCAFFE ROUND DOWN

Use CAFFE padding, rounding output size down, uses prePadding value.

kCAFFE ROUND UP

Use CAFFE padding, rounding output size up, uses prePadding value.

8.2.3.24

PluginFieldType

enum class nvinfer1::PluginFieldType : int32 t [strong]

The possible field types for custom layer.

Enumerator

kFLOAT16

FP16 field type.

kFLOAT32

FP32 field type.

kFLOAT64

FP64 field type.

kINT8

INT8 field type.

kINT16

INT16 field type.

kINT32

INT32 field type.

kCHAR

char field type.

kDIMS

nvinfer1::Dims field type.

kUNKNOWN

Unknown field type.

8.2.3.25

PluginVersion

enum class nvinfer1::PluginVersion : uint8 t [strong]

Enumerator

kV2

IPluginV2.

kV2 EXT

IPluginV2Ext.

kV2 IOEXT

IPluginV2IOExt.

kV2 DYNAMICEXT

IPluginV2DynamicExt.

8.2 nvinfer1 Namespace Reference

8.2.3.26

PoolingType

enum class nvinfer1::PoolingType : int32 t [strong]

The type of pooling to perform in a pooling layer.

Enumerator

kMAX

kAVERAGE

kMAX AVERAGE BLEND

8.2.3.27

PreviewFeature

enum class nvinfer1::PreviewFeature : int32 t [strong]

Define preview features.

Preview Features have been fully tested but are not yet as stable as other features in TensorRT. They are provided as

opt-in features for at least one release.

Enumerator

kFASTER DYNAMIC SHAPES 0805

Optimize runtime dimensions with TensorRT's DL

Compiler. Potentially reduces run time and decreases

device memory usage and engine size. Models most

likely to benefit from enabling

kFASTER DYNAMIC SHAPES 0805 are

transformer-based models, and models containing

dynamic control flows.

The default value for this flag is on.

Deprecated Turning it off is deprecated in TensorRT

8.6. The flag

kFASTER DYNAMIC SHAPES 0805

will be removed in 9.0.

kDISABLE EXTERNAL TACTIC SOURCES ←↩

Disable usage of cuDNN/cuBLAS/cuBLASLt tactics in

FOR CORE 0805

the TensorRT core library.

When the flag is enabled, TensorRT core will not use

these tactics even if they are specified in

IBuilderConfig::setTacticSources(), but cudnnContext

and cublasContext handles will still be passed to

plugins via IPluginV2Ext::attachToContext() if the

appropriate tactic sources are set.

This allows users to experiment with disabling external

library tactics without having to modify their

application's plugins to support nullptr handles.

The default value for this flag is on.

See also

TacticSource

Namespace Documentation

Enumerator

kPROFILE SHARING 0806

Allows optimization profiles to be shared across

execution contexts. This flag defaults to false and will

become the default behavior in TensorRT 9.0. At that

point this flag will do nothing.

8.2.3.28

ProfilingVerbosity

enum class nvinfer1::ProfilingVerbosity : int32 t [strong]

List of verbosity levels of layer information exposed in NVTX annotations and in IEngineInspector.

See also

IBuilderConfig::setProfilingVerbosity(), IBuilderConfig::getProfilingVerbosity(), IEngineInspector

Enumerator

kLAYER NAMES ONLY

Print only the layer names. This is the default setting.

kNONE

Do not print any layer information.

kDETAILED

Print detailed layer information including layer names and layer parameters.

kDEFAULT

Deprecated Deprecated in TensorRT 8.0. Superseded by kLAYER NAMES ONLY.

kVERBOSE

Deprecated Deprecated in TensorRT 8.0. Superseded by kDETAILED.

8.2.3.29

QuantizationFlag

enum class nvinfer1::QuantizationFlag : int32 t [strong]

List of valid flags for quantizing the network to int8.

See also

IBuilderConfig::setQuantizationFlag(), IBuilderConfig::getQuantizationFlag()

8.2 nvinfer1 Namespace Reference

Enumerator

kCALIBRATE BEFORE FUSION

Run int8 calibration pass before layer fusion. Only valid for

IInt8LegacyCalibrator and IInt8EntropyCalibrator. The builder always runs

the int8 calibration pass before layer fusion for IInt8MinMaxCalibrator and

IInt8EntropyCalibrator2. Disabled by default.

8.2.3.30

ReduceOperation

enum class nvinfer1::ReduceOperation : int32 t [strong]

Enumerates the reduce operations that may be performed by a Reduce layer.

The table shows the result of reducing across an empty volume of a given type.

Operation

kFLOAT and kHALF

kINT32

kINT8

kSUM

kPROD

kMAX

negative infinity

INT MIN

-128

kMIN

positive infinity

INT MAX

127

kAVG

NaN

-128

The current version of TensorRT usually performs reduction for kINT8 via kFLOAT or kHALF. The kINT8 values

show the quantized representations of the floating-point values.

Enumerator

kSUM

kPROD

kMAX

kMIN

kAVG

8.2.3.31

ResizeCoordinateTransformation

enum class nvinfer1::ResizeCoordinateTransformation : int32 t [strong]

The resize coordinate transformation function.

See also

IResizeLayer::setCoordinateTransformation()

Namespace Documentation

Enumerator

kALIGN CORNERS

Think of each value in the tensor as a unit volume, and the coordinate is a point inside

this volume. The coordinate point is drawn as a star (∗) in the below diagram, and

multiple values range has a length. Define x origin as the coordinate of axis x in the

input tensor, x resized as the coordinate of axis x in the output tensor,

length origin as length of the input tensor in axis x, and length resize as

length of the output tensor in axis x.

|<--------------length---------->|

x_origin = x_resized * (length_origin - 1) / (length_resize - 1)

kASYMMETRIC

|<------------length------------------->| | 0 | 1 | 2 | 3 |

x origin = x resized ∗ (length origin / length resize)

kHALF PIXEL

|<------------length------------------->| | 0 | 1 | 2 | 3 |

x origin = (x resized + 0.5) ∗ (length origin / length resize) - 0.5

8.2.3.32

ResizeRoundMode

enum class nvinfer1::ResizeRoundMode : int32 t [strong]

The rounding mode for nearest neighbor resize.

See also

IResizeLayer::setNearestRounding()

Enumerator

kHALF UP

Round half up.

kHALF DOWN

Round half down.

kFLOOR

Round to floor.

kCEIL

Round to ceil.

8.2.3.33

ResizeSelector

enum class nvinfer1::ResizeSelector : int32 t [strong]

The coordinate selector when resize to single pixel output.

See also

IResizeLayer::setSelectorForSinglePixel()

8.2 nvinfer1 Namespace Reference

Enumerator

kFORMULA

Use formula to map the original index.

kUPPER

Select the upper left pixel.

8.2.3.34

RNNDirection

enum class nvinfer1::RNNDirection : int32 t [strong]

Enumerates the RNN direction that may be performed by an RNN layer.

See also

IRNNv2Layer

Enumerator

kUNIDIRECTION

Network iterations from first input to last input.

kBIDIRECTION

Network iterates from first to last and vice versa and outputs concatenated.

8.2.3.35

RNNGateType

enum class nvinfer1::RNNGateType : int32 t [strong]

Identifies an individual gate within an RNN cell.

See also

RNNOperation

Enumerator

kINPUT

Input gate (i).

kOUTPUT

Output gate (o).

kFORGET

Forget gate (f).

kUPDATE

Update gate (z).

kRESET

Reset gate (r).

kCELL

Cell gate (c).

kHIDDEN

Hidden gate (h).

Namespace Documentation

8.2.3.36

RNNInputMode

enum class nvinfer1::RNNInputMode : int32 t [strong]

Enumerates the RNN input modes that may occur with an RNN layer.

If the RNN is configured with RNNInputMode::kLINEAR, then for each gate g in the first layer of the RNN, the input

vector X[t] (length E) is left-multiplied by the gate's corresponding weight matrix W[g] (dimensions HxE) as usual,

before being used to compute the gate output as described by RNNOperation.

If the RNN is configured with RNNInputMode::kSKIP, then this initial matrix multiplication is ”skipped” and W[g]

is conceptually an identity matrix. In this case, the input vector X[t] must have length H (the size of the hidden state).

See also

IRNNv2Layer

Enumerator

kLINEAR

Perform the normal matrix multiplication in the first recurrent layer.

kSKIP

No operation is performed on the first recurrent layer.

8.2.3.37

RNNOperation

enum class nvinfer1::RNNOperation : int32 t [strong]

Enumerates the RNN operations that may be performed by an RNN layer.

Equation definitions

The equations below have the following naming convention:

t := current time step

i := input gate

o := output gate

f := forget gate

z := update gate

r := reset gate

c := cell gate

h := hidden gate

g[t] denotes the output of gate g at timestep t, e.g.

f[t] is the output of the forget gate f.

X[t] := input tensor for timestep t

C[t] := cell state for timestep t

H[t] := hidden state for timestep t

W[g] := W (input) parameter weight matrix for gate g

R[g] := U (recurrent) parameter weight matrix for gate g

Wb[g] := W (input) parameter bias vector for gate g

Rb[g] := U (recurrent) parameter bias vector for gate g

Unless otherwise specified, all operations apply pointwise

to elements of each operand tensor.

8.2 nvinfer1 Namespace Reference

ReLU(X) := max(X, 0)

tanh(X) := hyperbolic tangent of X

sigmoid(X) := 1 / (1 + exp(-X))

exp(X) := eˆX

A.B denotes matrix multiplication of A and B.

A*B denotes pointwise multiplication of A and B.

Equations

Depending on the value of RNNOperation chosen, each sub-layer of the RNN layer will perform one of the following

operations:

::kRELU

H[t] := ReLU(W[i].X[t] + R[i].H[t-1] + Wb[i] + Rb[i])

::kTANH

H[t] := tanh(W[i].X[t] + R[i].H[t-1] + Wb[i] + Rb[i])

::kLSTM

i[t] := sigmoid(W[i].X[t] + R[i].H[t-1] + Wb[i] + Rb[i])

f[t] := sigmoid(W[f].X[t] + R[f].H[t-1] + Wb[f] + Rb[f])

o[t] := sigmoid(W[o].X[t] + R[o].H[t-1] + Wb[o] + Rb[o])

c[t] :=

tanh(W[c].X[t] + R[c].H[t-1] + Wb[c] + Rb[c])

C[t] := f[t]*C[t-1] + i[t]*c[t]

H[t] := o[t]*tanh(C[t])

::kGRU

z[t] := sigmoid(W[z].X[t] + R[z].H[t-1] + Wb[z] + Rb[z])

r[t] := sigmoid(W[r].X[t] + R[r].H[t-1] + Wb[r] + Rb[r])

h[t] := tanh(W[h].X[t] + r[t]*(R[h].H[t-1] + Rb[h]) + Wb[h])

H[t] := (1 - z[t])*h[t] + z[t]*H[t-1]

See also

IRNNv2Layer

Enumerator

kRELU

Single gate RNN w/ ReLU activation function.

kTANH

Single gate RNN w/ TANH activation function.

kLSTM

Four-gate LSTM network w/o peephole connections.

kGRU

Three-gate network consisting of Gated Recurrent Units.

8.2.3.38

SampleMode

enum class nvinfer1::SampleMode : int32 t [strong]

Controls how ISliceLayer and IGridSample handle out-of-bounds coordinates.

See also

ISliceLayer and IGridSample

Enumerator

kSTRICT BOUNDS

Fail with error when the coordinates are out of bounds.

kDEFAULT

Namespace Documentation

Enumerator

kWRAP

Coordinates wrap around periodically.

Deprecated Use kSTRICT BOUNDS.

kCLAMP

Out of bounds indices are clamped to bounds.

kFILL

Use fill input value when coordinates are out of bounds.

kREFLECT

Coordinates reflect. The axis of reflection is the middle of the perimeter pixel and the

reflections are repeated indefinitely within the padded regions. Repeats values for a single

pixel and throws error for zero pixels.

8.2.3.39

ScaleMode

enum class nvinfer1::ScaleMode : int32 t [strong]

Controls how shift, scale and power are applied in a Scale layer.

See also

IScaleLayer

Enumerator

kUNIFORM

Identical coefficients across all elements of the tensor.

kCHANNEL

Per-channel coefficients.

kELEMENTWISE

Elementwise coefficients.

8.2.3.40

ScatterMode

enum class nvinfer1::ScatterMode : int32 t [strong]

Control form of IScatterLayer.

See also

IScatterLayer

Enumerator

kELEMENT

Similar to ONNX ScatterElements.

kND

Similar to ONNX ScatterND.

8.2 nvinfer1 Namespace Reference

8.2.3.41

TacticSource

enum class nvinfer1::TacticSource : int32 t [strong]

List of tactic sources for TensorRT.

See also

TacticSources, IBuilderConfig::setTacticSources(), IBuilderConfig::getTacticSources(), PreviewFeature::kDISABLE EXTERNAL

Enumerator

kCUBLAS

cuBLAS tactics. Enabled by default.

Note

Disabling kCUBLAS will cause the cublas handle passed to plugins

in attachToContext to be null.

kCUBLAS LT

cuBLAS LT tactics. Enabled for x86 platforms and only enabled for

non-x86 platforms when CUDA >= 11.0 by default.

kCUDNN

cuDNN tactics. Enabled by default.

Note

Disabling kCUDNN will cause the cuDNN handle passed to plugins

in attachToContext to be null.

kEDGE MASK CONVOLUTIONS

Enables convolution tactics implemented with edge mask tables. These

tactics tradeoff memory for performance by consuming additional

memory space proportional to the input size. Enabled by default.

kJIT CONVOLUTIONS

Enables convolution tactics implemented with source-code JIT fusion.

The engine building time may increase when this is enabled. Enabled by

default.

8.2.3.42

TempfileControlFlag

enum class nvinfer1::TempfileControlFlag : int32 t [strong]

Flags used to control TensorRT's behavior when creating executable temporary files.

On some platforms the TensorRT runtime may need to create files in a temporary directory or use platform-specific

APIs to create files in-memory to load temporary DLLs that implement runtime code. These flags allow the application

to explicitly control TensorRT's use of these files. This will preclude the use of certain TensorRT APIs for deserializing

and loading lean runtimes.

Namespace Documentation

Enumerator

kALLOW IN MEMORY FILES

Allow creating and loading files in-memory (or unnamed files).

kALLOW TEMPORARY FILES

Allow creating and loading named files in a temporary directory on the

filesystem.

\see IRuntime::setTemporaryDirectory()

8.2.3.43

TensorFormat

enum class nvinfer1::TensorFormat : int32 t [strong]

Format of the input/output tensors.

This enum is used by both plugins and network I/O tensors.

See also

IPluginV2::supportsFormat(), safe::ICudaEngine::getBindingFormat()

For more information about data formats, see the topic ”Data Format Description” located in the TensorRT Developer

Guide.

Enumerator

kLINEAR

Row major linear format. For a tensor with dimensions {N, C, H, W} or {numbers, channels,

columns, rows}, the dimensional index corresponds to {3, 2, 1, 0} and thus the order is W

minor.

For DLA usage, the tensor sizes are limited to C,H,W in the range [1,8192].

kCHW2

Two wide channel vectorized row major format. This format is bound to FP16. It is only

available for dimensions >= 3. For a tensor with dimensions {N, C, H, W}, the memory

layout is equivalent to a C array with dimensions [N][(C+1)/2][H][W][2], with the tensor

coordinates (n, c, h, w) mapping to array subscript [n][c/2][h][w][c%2].

kHWC8

Eight channel format where C is padded to a multiple of 8. This format is bound to FP16. It is

only available for dimensions >= 3. For a tensor with dimensions {N, C, H, W}, the memory

layout is equivalent to the array with dimensions [N][H][W][(C+7)/8∗8], with the tensor

coordinates (n, c, h, w) mapping to array subscript [n][h][w][c].

kCHW4

Four wide channel vectorized row major format. This format is bound to INT8 or FP16. It is

only available for dimensions >= 3. For INT8, the C dimension must be a build-time constant.

For a tensor with dimensions {N, C, H, W}, the memory layout is equivalent to a C array with

dimensions [N][(C+3)/4][H][W][4], with the tensor coordinates (n, c, h, w) mapping to array

subscript [n][c/4][h][w][c%4].

Deprecated usage:

If running on the DLA, this format can be used for acceleration with the caveat that C must be

equal or lesser than 4. If used as DLA input and the build option kGPU FALLBACK is not

specified, it needs to meet line stride requirement of DLA format. Column stride in bytes

should be a multiple of 32 on Xavier and 64 on Orin.

8.2 nvinfer1 Namespace Reference

Enumerator

kCHW16

Sixteen wide channel vectorized row major format. This format is bound to FP16. It is only

available for dimensions >= 3. For a tensor with dimensions {N, C, H, W}, the memory

layout is equivalent to a C array with dimensions [N][(C+15)/16][H][W][16], with the tensor

coordinates (n, c, h, w) mapping to array subscript [n][c/16][h][w][c%16].

For DLA usage, this format maps to the native feature format for FP16, and the tensor sizes

are limited to C,H,W in the range [1,8192].

kCHW32

Thirty-two wide channel vectorized row major format. This format is only available for

dimensions >= 3. For a tensor with dimensions {N, C, H, W}, the memory layout is

equivalent to a C array with dimensions [N][(C+31)/32][H][W][32], with the tensor

coordinates (n, c, h, w) mapping to array subscript [n][c/32][h][w][c%32].

For DLA usage, this format maps to the native feature format for INT8, and the tensor sizes

are limited to C,H,W in the range [1,8192].

kDHWC8

Eight channel format where C is padded to a multiple of 8. This format is bound to FP16, and

it is only available for dimensions >= 4. For a tensor with dimensions {N, C, D, H, W}, the

memory layout is equivalent to an array with dimensions [N][D][H][W][(C+7)/8∗8], with the

tensor coordinates (n, c, d, h, w) mapping to array subscript [n][d][h][w][c].

kCDHW32

Thirty-two wide channel vectorized row major format. This format is bound to FP16 and INT8

and is only available for dimensions >= 4. For a tensor with dimensions {N, C, D, H, W}, the

memory layout is equivalent to a C array with dimensions [N][(C+31)/32][D][H][W][32], with

the tensor coordinates (n, c, d, h, w) mapping to array subscript [n][c/32][d][h][w][c%32].

kHWC

Non-vectorized channel-last format. This format is bound to either FP32 or UINT8, and is

only available for dimensions >= 3.

kDLA LINEAR

DLA planar format. For a tensor with dimension {N, C, H, W}, the W axis always has unit

stride. The stride for stepping along the H axis is rounded up to 64 bytes.

The memory layout is equivalent to a C array with dimensions [N][C][H][roundUp(W,

64/elementSize)] where elementSize is 2 for FP16 and 1 for Int8, with the tensor coordinates

(n, c, h, w) mapping to array subscript [n][c][h][w].

kDLA HWC4

DLA image format. For a tensor with dimension {N, C, H, W} the C axis always has unit

stride. The stride for stepping along the H axis is rounded up to 32 bytes on Xavier and 64

bytes on Orin. C can only be 1, 3 or 4. If C == 1, it will map to grayscale format. If C == 3 or

C == 4, it will map to color image format. And if C == 3, the stride for stepping along the W

axis needs to be padded to 4 in elements.

When C is {1, 3, 4}, then C' is {1, 4, 4} respectively, the memory layout is equivalent to a C

array with dimensions [N][H][roundUp(W, 32/C'/elementSize)][C'] on Xavier and

[N][H][roundUp(W, 64/C'/elementSize)][C'] on Orin where elementSize is 2 for FP16 and 1

for Int8. The tensor coordinates (n, c, h, w) mapping to array subscript [n][h][w][c].

kHWC16

Sixteen channel format where C is padded to a multiple of 16. This format is bound to FP16.

It is only available for dimensions >= 3. For a tensor with dimensions {N, C, H, W}, the

memory layout is equivalent to the array with dimensions [N][H][W][(C+15)/16∗16], with the

tensor coordinates (n, c, h, w) mapping to array subscript [n][h][w][c].

kDHWC

Non-vectorized channel-last format. This format is bound to FP32. It is only available for

dimensions >= 4.

8.2.3.44

TensorIOMode

enum class nvinfer1::TensorIOMode : int32 t [strong]

8.2 nvinfer1 Namespace Reference

Enumerator

kNONE

Tensor is not an input or output.

kINPUT

Tensor is input to the engine.

kOUTPUT

Tensor is output by the engine.

8.2.3.45

TensorLocation

enum class nvinfer1::TensorLocation : int32 t [strong]

The location for tensor data storage, device or host.

Enumerator

kDEVICE

Data stored on device.

kHOST

Data stored on host.

8.2.3.46

TopKOperation

enum class nvinfer1::TopKOperation : int32 t [strong]

Enumerates the operations that may be performed by a TopK layer.

Enumerator

kMAX

Maximum of the elements.

kMIN

Minimum of the elements.

8.2.3.47

TripLimit

enum class nvinfer1::TripLimit : int32 t [strong]

Enum that describes kinds of trip limits.

Enumerator

kCOUNT

Tensor is scalar of type kINT32 that contains the trip count.

kWHILE

Tensor is a scalar of type kBOOL. Loop terminates when value is false.

Namespace Documentation

8.2.3.48

UnaryOperation

enum class nvinfer1::UnaryOperation : int32 t [strong]

Enumerates the unary operations that may be performed by a Unary layer.

Operations kNOT must have inputs of DataType::kBOOL.

Operation kSIGN must have inputs of DataType::kFLOAT, DataType::kHALF, DataType::kINT8, or DataType::kINT32.

Operation kISINF must have inputs of DataType::kFLOAT or DataType::kHALF.

All other operations must have inputs of DataType::kFLOAT, DataType::kHALF, or DataType::kINT8.

Operations kSIGN and kROUND are not supported in implicit batch mode.

See also

IUnaryLayer

Enumerator

kEXP

Exponentiation.

kLOG

Log (base e).

kSQRT

Square root.

kRECIP

Reciprocal.

kABS

Absolute value.

kNEG

Negation.

kSIN

Sine.

kCOS

Cosine.

kTAN

Tangent.

kSINH

Hyperbolic sine.

kCOSH

Hyperbolic cosine.

kASIN

Inverse sine.

kACOS

Inverse cosine.

kATAN

Inverse tangent.

kASINH

Inverse hyperbolic sine.

kACOSH

Inverse hyperbolic cosine.

kATANH

Inverse hyperbolic tangent.

kCEIL

Ceiling.

kFLOOR

Floor.

kERF

Gauss error function.

kNOT

Logical NOT.

kSIGN

Sign, If input > 0, output 1; if input < 0, output -1; if input == 0, output 0.

kROUND

Round to nearest even for floating-point data type.

kISINF

Return true if input value equals +/- infinity for floating-point data type.

8.2 nvinfer1 Namespace Reference

8.2.3.49

WeightsRole

enum class nvinfer1::WeightsRole : int32 t [strong]

How a layer uses particular Weights.

The power weights of an IScaleLayer are omitted. Refitting those is not supported.

Enumerator

kKERNEL

kernel for IConvolutionLayer, IDeconvolutionLayer, or IFullyConnectedLayer

kBIAS

bias for IConvolutionLayer, IDeconvolutionLayer, or IFullyConnectedLayer

kSHIFT

shift part of IScaleLayer

kSCALE

scale part of IScaleLayer

kCONSTANT

weights for IConstantLayer

kANY

Any other weights role.

8.2.4

Function Documentation

8.2.4.1

EnumMax()

template<typename T >

constexpr int32 t nvinfer1::EnumMax ( ) [constexpr], [noexcept]

Maximum number of elements in an enumeration type.

8.2.4.2

EnumMax< BoundingBoxFormat >()

template<>

constexpr int32 t nvinfer1::EnumMax< BoundingBoxFormat > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in BoundingBoxFormat enum.

See also

BoundingBoxFormat

Namespace Documentation

8.2.4.3

EnumMax< BuilderFlag >()

template<>

constexpr int32 t nvinfer1::EnumMax< BuilderFlag > ( ) [inline], [constexpr], [noexcept]

Maximum number of builder flags in BuilderFlag enum.

See also

BuilderFlag

8.2.4.4

EnumMax< CalibrationAlgoType >()

template<>

constexpr int32 t nvinfer1::EnumMax< CalibrationAlgoType > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in CalibrationAlgoType enum.

See also

DataType

8.2.4.5

EnumMax< DeviceType >()

template<>

constexpr int32 t nvinfer1::EnumMax< DeviceType > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in DeviceType enum.

See also

DeviceType

8.2.4.6

EnumMax< DimensionOperation >()

template<>

constexpr int32 t nvinfer1::EnumMax< DimensionOperation > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in DimensionOperation enum.

See also

DimensionOperation

8.2 nvinfer1 Namespace Reference

8.2.4.7

EnumMax< FillOperation >()

template<>

constexpr int32 t nvinfer1::EnumMax< FillOperation > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in FillOperation enum.

See also

FillOperation

8.2.4.8

EnumMax< GatherMode >()

template<>

constexpr int32 t nvinfer1::EnumMax< GatherMode > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in GatherMode enum.

See also

GatherMode

8.2.4.9

EnumMax< LayerInformationFormat >()

template<>

constexpr int32 t nvinfer1::EnumMax< LayerInformationFormat > ( ) [inline], [constexpr], [noexcept]

Maximum number of layer information formats in LayerInformationFormat enum.

See also

LayerInformationFormat

8.2.4.10

EnumMax< LayerType >()

template<>

constexpr int32 t nvinfer1::EnumMax< LayerType > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in LayerType enum.

See also

LayerType

Namespace Documentation

8.2.4.11

EnumMax< LoopOutput >()

template<>

constexpr int32 t nvinfer1::EnumMax< LoopOutput > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in LoopOutput enum.

See also

DataType

8.2.4.12

EnumMax< MatrixOperation >()

template<>

constexpr int32 t nvinfer1::EnumMax< MatrixOperation > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in MatrixOperation enum.

See also

DataType

8.2.4.13

EnumMax< MemoryPoolType >()

template<>

constexpr int32 t nvinfer1::EnumMax< MemoryPoolType > ( ) [inline], [constexpr], [noexcept]

Maximum number of memory pool types in the MemoryPoolType enum.

See also

MemoryPoolType

8.2.4.14

EnumMax< NetworkDefinitionCreationFlag >()

template<>

constexpr int32 t nvinfer1::EnumMax< NetworkDefinitionCreationFlag > ( ) [inline], [constexpr],

[noexcept]

Maximum number of elements in NetworkDefinitionCreationFlag enum.

See also

NetworkDefinitionCreationFlag

8.2 nvinfer1 Namespace Reference

8.2.4.15

EnumMax< OptProfileSelector >()

template<>

constexpr int32 t nvinfer1::EnumMax< OptProfileSelector > ( ) [inline], [constexpr], [noexcept]

Number of different values of OptProfileSelector enum.

See also

OptProfileSelector

8.2.4.16

EnumMax< ProfilingVerbosity >()

template<>

constexpr int32 t nvinfer1::EnumMax< ProfilingVerbosity > ( ) [inline], [constexpr], [noexcept]

Maximum number of profile verbosity levels in ProfilingVerbosity enum.

See also

ProfilingVerbosity

8.2.4.17

EnumMax< QuantizationFlag >()

template<>

constexpr int32 t nvinfer1::EnumMax< QuantizationFlag > ( ) [inline], [constexpr], [noexcept]

Maximum number of quantization flags in QuantizationFlag enum.

See also

QuantizationFlag

8.2.4.18

EnumMax< ReduceOperation >()

template<>

constexpr int32 t nvinfer1::EnumMax< ReduceOperation > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in ReduceOperation enum.

See also

ReduceOperation

Namespace Documentation

8.2.4.19

EnumMax< RNNDirection >()

template<>

constexpr int32 t nvinfer1::EnumMax< RNNDirection > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in RNNDirection enum.

See also

RNNDirection

8.2.4.20

EnumMax< RNNGateType >()

template<>

constexpr int32 t nvinfer1::EnumMax< RNNGateType > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in RNNGateType enum.

See also

RNNGateType

8.2.4.21

EnumMax< RNNInputMode >()

template<>

constexpr int32 t nvinfer1::EnumMax< RNNInputMode > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in RNNInputMode enum.

See also

RNNInputMode

8.2.4.22

EnumMax< RNNOperation >()

template<>

constexpr int32 t nvinfer1::EnumMax< RNNOperation > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in RNNOperation enum.

See also

RNNOperation

8.2 nvinfer1 Namespace Reference

8.2.4.23

EnumMax< SampleMode >()

template<>

constexpr int32 t nvinfer1::EnumMax< SampleMode > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in SampleMode enum.

See also

SampleMode

8.2.4.24

EnumMax< ScaleMode >()

template<>

constexpr int32 t nvinfer1::EnumMax< ScaleMode > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in ScaleMode enum.

See also

ScaleMode

8.2.4.25

EnumMax< ScatterMode >()

template<>

constexpr int32 t nvinfer1::EnumMax< ScatterMode > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in ScatterMode enum.

See also

ScatterMode

8.2.4.26

EnumMax< TacticSource >()

template<>

constexpr int32 t nvinfer1::EnumMax< TacticSource > ( ) [inline], [constexpr], [noexcept]

Maximum number of tactic sources in TacticSource enum.

See also

TacticSource

Namespace Documentation

8.2.4.27

EnumMax< TempfileControlFlag >()

template<>

constexpr int32 t nvinfer1::EnumMax< TempfileControlFlag > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in TempfileControlFlag enum.

See also

TempfileControlFlag

8.2.4.28

EnumMax< TopKOperation >()

template<>

constexpr int32 t nvinfer1::EnumMax< TopKOperation > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in TopKOperation enum.

See also

TopKOperation

8.2.4.29

EnumMax< TripLimit >()

template<>

constexpr int32 t nvinfer1::EnumMax< TripLimit > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in TripLimit enum.

See also

DataType

8.2.4.30

EnumMax< UnaryOperation >()

template<>

constexpr int32 t nvinfer1::EnumMax< UnaryOperation > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in UnaryOperation enum.

See also

UnaryOperation

8.3 nvinfer1::consistency Namespace Reference

8.2.4.31

EnumMax< WeightsRole >()

template<>

constexpr int32 t nvinfer1::EnumMax< WeightsRole > ( ) [inline], [constexpr], [noexcept]

Maximum number of elements in WeightsRole enum.

See also

WeightsRole

8.2.4.32

getBuilderPluginRegistry()

nvinfer1::IPluginRegistry ∗ nvinfer1::getBuilderPluginRegistry (

nvinfer1::EngineCapability capability ) [noexcept]

Return the plugin registry for building a Standard engine, or nullptr if no registry exists.

Also return nullptr if the input argument is not EngineCapability::kSTANDARD. Engine capabilities EngineCapability::kSTANDARD

and EngineCapability::kSAFETY have distinct plugin registries. When building a Safety engine, use nvinfer1::getBuilderSafePluginRegi

Use IPluginRegistry::registerCreator from the registry to register plugins. Plugins registered in a registry associated

with a specific engine capability are only available when building engines with that engine capability.

There is no plugin registry for EngineCapability::kDLA STANDALONE.

8.2.4.33

getBuilderSafePluginRegistry()

nvinfer1::safe::IPluginRegistry ∗ nvinfer1::getBuilderSafePluginRegistry (

nvinfer1::EngineCapability capability ) [noexcept]

Return the plugin registry for building a Safety engine, or nullptr if no registry exists.

Also return nullptr if the input argument is not EngineCapability::kSAFETY. When building a Standard engine, use

nvinfer1::getBuilderPluginRegistry(). Use safe::IPluginRegistry::registerCreator from the registry to register plugins.

8.3

nvinfer1::consistency Namespace Reference

Classes

• class IConsistencyChecker

Validates a serialized engine blob.

• class IPluginChecker

Consistency Checker plugin class for user implemented Plugins.

Namespace Documentation

8.4

nvinfer1::impl Namespace Reference

Classes

•

struct EnumMaxImpl

Declaration of EnumMaxImpl struct to store maximum number of elements in an enumeration type.

•

struct EnumMaxImpl< ActivationType >

•

struct EnumMaxImpl< AllocatorFlag >

Maximum number of elements in AllocatorFlag enum.

•

struct EnumMaxImpl< DataType >

Maximum number of elements in DataType enum.

•

struct EnumMaxImpl< ElementWiseOperation >

•

struct EnumMaxImpl< EngineCapability >

Maximum number of elements in EngineCapability enum.

•

struct EnumMaxImpl< ErrorCode >

Maximum number of elements in ErrorCode enum.

•

struct EnumMaxImpl< HardwareCompatibilityLevel >

•

struct EnumMaxImpl< ILogger::Severity >

Maximum number of elements in ILogger::Severity enum.

•

struct EnumMaxImpl< InterpolationMode >

•

struct EnumMaxImpl< PaddingMode >

•

struct EnumMaxImpl< PoolingType >

•

struct EnumMaxImpl< PreviewFeature >

•

struct EnumMaxImpl< ResizeCoordinateTransformation >

•

struct EnumMaxImpl< ResizeRoundMode >

•

struct EnumMaxImpl< ResizeSelector >

•

struct EnumMaxImpl< TensorFormat >

Maximum number of elements in TensorFormat enum.

•

struct EnumMaxImpl< TensorIOMode >

Maximum number of elements in TensorIOMode enum.

•

struct EnumMaxImpl< TensorLocation >

Maximum number of elements in TensorLocation enum.

8.5

nvinfer1::plugin Namespace Reference

Classes

• struct DetectionOutputParameters

The DetectionOutput plugin layer generates the detection output based on location and confidence predictions by doing

non maximum suppression. This plugin first decodes the bounding boxes based on the anchors generated. It then

performs non max suppression on the decoded bounding boxes. DetectionOutputParameters defines a set of parameters

for creating the DetectionOutput plugin layer. It contains:

• struct GridAnchorParameters

The Anchor Generator plugin layer generates the prior boxes of designated sizes and aspect ratios across all dimen-

sions (H x W). GridAnchorParameters defines a set of parameters for creating the plugin layer for all feature maps. It

contains:

• struct NMSParameters

8.5 nvinfer1::plugin Namespace Reference

The NMSParameters are used by the BatchedNMSPlugin for performing the non max suppression operation over boxes

for object detection networks.

• struct PriorBoxParameters

The PriorBox plugin layer generates the prior boxes of designated sizes and aspect ratios across all dimensions (H x W).

PriorBoxParameters defines a set of parameters for creating the PriorBox plugin layer. It contains:

• struct Quadruple

The Permute plugin layer permutes the input tensor by changing the memory order of the data. Quadruple defines a

structure that contains an array of 4 integers. They can represent the permute orders or the strides in each dimension.

• struct RegionParameters

The Region plugin layer performs region proposal calculation: generate 5 bounding boxes per cell (for yolo9000, gen-

erate 3 bounding boxes per cell). For each box, calculating its probablities of objects detections from 80 pre-defined

classifications (yolo9000 has 9418 pre-defined classifications, and these 9418 items are organized as work-tree struc-

ture). RegionParameters defines a set of parameters for creating the Region plugin layer.

• struct RPROIParams

RPROIParams is used to create the RPROIPlugin instance. It contains:

• struct softmaxTree

When performing yolo9000, softmaxTree is helping to do softmax on confidence scores, for element to get the precise

classification through word-tree structured classification definition.

Enumerations

• enum class CodeTypeSSD : int32 t { CORNER = 0 , CENTER SIZE = 1 , CORNER SIZE = 2 , TF CENTER

=3}

The type of encoding used for decoding the bounding boxes and loc data.

8.5.1

Enumeration Type Documentation

8.5.1.1

CodeTypeSSD

enum class nvinfer1::plugin::CodeTypeSSD : int32 t [strong]

The type of encoding used for decoding the bounding boxes and loc data.

Enumerator

CORNER

Use box corners.

CENTER SIZE

Use box centers and size.

CORNER SIZE

Use box centers and size.

TF CENTER

Use box centers and size but flip x and y coordinates.

Namespace Documentation

8.6

nvinfer1::safe Namespace Reference

The safety subset of TensorRT's API version 1 namespace.

Classes

• class ICudaEngine

A functionally safe engine for executing inference on a built network.

• class IExecutionContext

Functionally safe context for executing inference using an engine.

• class IPluginRegistry

Single registration point for all plugins in an application. It is used to find plugin implementations during engine dese-

rialization. Internally, the plugin registry is considered to be a singleton so all plugins in an application are part of the

same global registry. Note that the plugin registry is only supported for plugins of type IPluginV2 and should also have

a corresponding IPluginCreator implementation.

• class IRuntime

Allows a serialized functionally safe engine to be deserialized.

• class PluginRegistrar

Register the plugin creator to the registry The static registry object will be instantiated when the plugin library is loaded.

This static object will register all creators available in the library to the registry.

• struct RuntimeErrorInformation

Space to record information about runtime errors.

Enumerations

• enum class RuntimeErrorType : uint64 t {

kNAN CONSUMED = 1ULL << 0 , kINF CONSUMED = 1ULL << 1 , kGATHER OOB = 1ULL << 2 ,

kSCATTER OOB = 1ULL << 3 ,

kSCATTER RACE = 1ULL << 4 }

Functions

• IRuntime ∗ createInferRuntime (ILogger &logger) noexcept

Create an instance of an safe::IRuntime class.

• IPluginRegistry ∗ getSafePluginRegistry () noexcept

Return the safe plugin registry.

8.6.1

Detailed Description

The safety subset of TensorRT's API version 1 namespace.

8.6.2

Enumeration Type Documentation

8.6.2.1

RuntimeErrorType

enum class nvinfer1::safe::RuntimeErrorType : uint64 t [strong]

8.6 nvinfer1::safe Namespace Reference

Enumerator

kNAN CONSUMED

NaN floating-point value was silently consumed.

kINF CONSUMED

Inf floating-point value was silently consumed.

kGATHER OOB

Out-of-bounds access in gather operation.

kSCATTER OOB

Out-of-bounds access in scatter operation.

kSCATTER RACE

Race condition in scatter operation.

8.6.3

Function Documentation

8.6.3.1

createInferRuntime()

IRuntime ∗ nvinfer1::safe::createInferRuntime (

ILogger & logger ) [noexcept]

Create an instance of an safe::IRuntime class.

This class is the logging class for the runtime.

Usage considerations

• Allowed context for the API call

- Thread-safe: Yes

8.6.3.2

getSafePluginRegistry()

IPluginRegistry ∗ nvinfer1::safe::getSafePluginRegistry ( ) [noexcept]

Return the safe plugin registry.

Usage considerations

• Allowed context for the API call

- Thread-safe: Yes

Namespace Documentation

8.7

nvinfer1::utils Namespace Reference

Functions

• TRT DEPRECATED bool reshapeWeights (Weights const &input, int32 t const ∗shape, int32 t const ∗shape←↩

Order, void ∗data, int32 t nbDims) noexcept

Reformat the input weights of the given shape based on the new order of dimensions.

• TRT DEPRECATED bool reorderSubBuffers (void ∗input, int32 t const ∗order, int32 t num, int32 t size) noex-

cept

Takes an input stream and re-orders num chunks of the data given the size and order.

• TRT DEPRECATED bool transposeSubBuffers (void ∗input, DataType type, int32 t num, int32 t height, int32←↩

t width) noexcept

Transpose num sub-buffers of height ∗ width.

8.7.1

Function Documentation

8.7.1.1

reorderSubBuffers()

TRT DEPRECATED bool nvinfer1::utils::reorderSubBuffers (

void ∗ input,

int32 t const ∗ order,

int32 t num,

int32 t size ) [noexcept]

Takes an input stream and re-orders num chunks of the data given the size and order.

Parameters

input

The input data to re-order.

order

The new order of the data sub-buffers.

num

The number of data sub-buffers to re-order.

size

The size of each data sub-buffer in bytes.

In some frameworks, the ordering of the sub-buffers within a dimension is different than the way that TensorRT expects

them. TensorRT expects the gate/bias sub-buffers for LSTM's to be in fico order. TensorFlow however formats the sub-

buffers in icfo order. This helper function solves this in a generic fashion.

Example usage output of reshapeWeights above: int32 t indir[1]{1, 0} int32 t stride = W∗H; for (int32 t x = 0, y =

N∗C; x < y; ++x) reorderSubBuffers(out + x ∗ stride, indir, H, W);

Input Matrix{2, 3, 2, 3}: { 0 2 4}, { 1 3 5} <- {0, 0, ∗, ∗} {12 14 16}, {13 15 17} <- {0, 1, ∗, ∗} {24 26 28}, {25 27

29} <- {0, 2, ∗, ∗} { 6 8 10}, { 7 9 11} <- {1, 0, ∗, ∗} {18 20 22}, {19 21 23} <- {1, 1, ∗, ∗} {30 32 34}, {31 33

35} <- {1, 2, ∗, ∗}

Output Matrix{2, 3, 2, 3}: { 1 3 5}, { 0 2 4} <- {0, 0, ∗, ∗} {13 15 17}, {12 14 16} <- {0, 1, ∗, ∗} {25 27 29}, {24

26 28} <- {0, 2, ∗, ∗} { 7 9 11}, { 6 8 10} <- {1, 0, ∗, ∗} {19 21 23}, {18 20 22} <- {1, 1, ∗, ∗} {31 33 35}, {30

32 34} <- {1, 2, ∗, ∗}

8.7 nvinfer1::utils Namespace Reference

Returns

True on success, false on failure.

See also

reshapeWeights()

Deprecated Deprecated in TensorRT 8.0.

Warning

This file will be removed in TensorRT 10.0.

8.7.1.2

reshapeWeights()

TRT DEPRECATED bool nvinfer1::utils::reshapeWeights (

Weights const & input,

int32 t const ∗ shape,

int32 t const ∗ shapeOrder,

void ∗ data,

int32 t nbDims ) [noexcept]

Reformat the input weights of the given shape based on the new order of dimensions.

Parameters

input

The input weights to reshape.

shape

The shape of the weights.

shapeOrder

The order of the dimensions to process for the output.

data

The location where the output data is placed.

nbDims

The number of dimensions to process.

Take the weights specified by input with the dimensions specified by shape and re-order the weights based on the

new dimensions specified by shapeOrder. The size of each dimension and the input data is not modified. The output

volume pointed to by data must be the same as he input volume.

Example usage: float ∗out = new float[N∗C∗H∗W]; Weights input{DataType::kFLOAT, {0 ... N∗C∗H∗W-1}, N∗←↩

C∗H∗W size}; int32 t order[4]{1, 0, 3, 2}; int32 t shape[4]{C, N, W, H}; reshapeWeights(input, shape, order, out, 4);

Weights reshaped{input.type, out, input.count};

Input Matrix{3, 2, 3, 2}: { 0 1}, { 2 3}, { 4 5} <- {0, 0, ∗, ∗} { 6 7}, { 8 9}, {10 11} <- {0, 1, ∗, ∗} {12 13}, {14

15}, {16 17} <- {1, 0, ∗, ∗} {18 19}, {20 21}, {22 23} <- {1, 1, ∗, ∗} {24 25}, {26 27}, {28 29} <- {2, 0, ∗, ∗}

{30 31}, {32 33}, {34 35} <- {2, 1, ∗, ∗}

Output Matrix{2, 3, 2, 3}: { 0 2 4}, { 1 3 5} <- {0, 0, ∗, ∗} {12 14 16}, {13 15 17} <- {0, 1, ∗, ∗} {24 26 28}, {25

27 29} <- {0, 2, ∗, ∗} { 6 8 10}, { 7 9 11} <- {1, 0, ∗, ∗} {18 20 22}, {19 21 23} <- {1, 1, ∗, ∗} {30 32 34}, {31

33 35} <- {1, 2, ∗, ∗}

Namespace Documentation

Returns

True on success, false on failure.

Deprecated Deprecated in TensorRT 8.0.

Warning

This file will be removed in TensorRT 10.0.

8.7.1.3

transposeSubBuffers()

TRT DEPRECATED bool nvinfer1::utils::transposeSubBuffers (

void ∗ input,

DataType type,

int32 t num,

int32 t height,

int32 t width ) [noexcept]

Transpose num sub-buffers of height ∗ width.

Parameters

input

The input data to transpose.

type

The type of the data to transpose.

num

The number of data sub-buffers to transpose.

height

The size of the height dimension to transpose.

width

The size of the width dimension to transpose.

Returns

True on success, false on failure.

Deprecated Deprecated in TensorRT 8.0.

Warning

This file will be removed in TensorRT 10.0.

8.8

nvonnxparser Namespace Reference

The TensorRT ONNX parser API namespace.

8.8 nvonnxparser Namespace Reference

Classes

• class IOnnxConfig

Configuration Manager Class.

• class IParser

an object for parsing ONNX models into a TensorRT network definition

• class IParserError

an object containing information about an error

Typedefs

• using OnnxParserFlags = uint32 t

Represents one or more OnnxParserFlag values using binary OR operations, e.g., 1U << OnnxParserFlag::kNATIVE INSTANCENORM.

Enumerations

• enum class ErrorCode : int {

kSUCCESS = 0 , kINTERNAL ERROR = 1 , kMEM ALLOC FAILED = 2 , kMODEL DESERIALIZE FAILED

=3,

kINVALID VALUE = 4 , kINVALID GRAPH = 5 , kINVALID NODE = 6 , kUNSUPPORTED GRAPH = 7 ,

kUNSUPPORTED NODE = 8 }

The type of error that the parser may return.

• enum class OnnxParserFlag : int32 t { kNATIVE INSTANCENORM = 0 }

Functions

• IOnnxConfig ∗ createONNXConfig ()

• template<typename T >

constexpr int32 t EnumMax ()

• template<> constexpr int32 t EnumMax< ErrorCode > ()

• template<> constexpr int32 t EnumMax< OnnxParserFlag > ()

8.8.1

Detailed Description

The TensorRT ONNX parser API namespace.

8.8.2

Typedef Documentation

8.8 nvonnxparser Namespace Reference

Enumerator

kNATIVE INSTANCENORM

Parse the ONNX model into the INetworkDefinition with the intention of using

TensorRT's native layer implementation over the plugin implementation for

InstanceNormalization nodes. This flag is planned to be deprecated in TensorRT

8.7 and removed in TensorRT 9.0. This flag is required when building

version-compatible or hardware-compatible engines. There may be performance

degradations when this flag is enabled.

8.8.4

Function Documentation

8.8.4.1

createONNXConfig()

IOnnxConfig ∗ nvonnxparser::createONNXConfig ( )

8.8.4.2

EnumMax()

template<typename T >

constexpr int32 t nvonnxparser::EnumMax ( ) [inline], [constexpr]

8.8.4.3

EnumMax< ErrorCode >()

template<>

constexpr int32 t nvonnxparser::EnumMax< ErrorCode > ( ) [inline], [constexpr]

Maximum number of flags in the ErrorCode enum.

See also

ErrorCode

Namespace Documentation

8.8.4.4

EnumMax< OnnxParserFlag >()

template<>

constexpr int32 t nvonnxparser::EnumMax< OnnxParserFlag > ( ) [inline], [constexpr]

Maximum number of flags in the OnnxParserFlag enum.

See also

OnnxParserFlag

8.9

nvuffparser Namespace Reference

The TensorRT UFF parser API namespace.

Classes

• struct FieldCollection

• class FieldMap

An array of field params used as a layer parameter for plugin layers.

• class IUffParser

Class used for parsing models described using the UFF format.

Enumerations

• enum class UffInputOrder : int32 t { kNCHW = 0 , kNHWC = 1 , kNC = 2 }

The different possible supported input order.

• enum class FieldType : int32 t {

kFLOAT = 0 , kINT32 = 1 , kCHAR = 2 , kDIMS = 4 ,

kDATATYPE = 5 , kUNKNOWN = 6 }

The possible field types for custom layer.

Functions

• IUffParser ∗ createUffParser () noexcept

Creates a IUffParser object.

• void shutdownProtobufLibrary (void) noexcept

Shuts down protocol buffers library.

8.9.1

Detailed Description

The TensorRT UFF parser API namespace.

Namespace Documentation

Enumerator

kFLOAT

FP32 field type.

kINT32

INT32 field type.

kCHAR

char field type. String for length>1.

kDIMS

nvinfer1::Dims field type.

kDATATYPE

nvinfer1::DataType field type.

kUNKNOWN

8.9.2.2

UffInputOrder

enum class nvuffparser::UffInputOrder : int32 t [strong]

The different possible supported input order.

Enumerator

kNCHW

NCHW order.

kNHWC

NHWC order.

kNC

NC order.

8.9.3

Function Documentation

8.9.3.1

createUffParser()

IUffParser ∗ nvuffparser::createUffParser ( ) [noexcept]

Creates a IUffParser object.

Returns

A pointer to the IUffParser object is returned.

See also

nvuffparser::IUffParser

Deprecated IUffParser will be removed in TensorRT 9.0. Plan to migrate your workflow to use nvonnxparser::IParser

for deployment.

Chapter 9

Class Documentation

9.1

nvinfer1::plugin::DetectionOutputParameters Struct Reference

The DetectionOutput plugin layer generates the detection output based on location and confidence predictions by

doing non maximum suppression. This plugin first decodes the bounding boxes based on the anchors generated. It

then performs non max suppression on the decoded bounding boxes. DetectionOutputParameters defines a set of

parameters for creating the DetectionOutput plugin layer. It contains:

#include <NvInferPluginUtils.h>

Public Attributes

• bool shareLocation

• bool varianceEncodedInTarget

• int32 t backgroundLabelId

• int32 t numClasses

• int32 t topK

• int32 t keepTopK

• float confidenceThreshold

• float nmsThreshold

• CodeTypeSSD codeType

• int32 t inputOrder [3]

• bool confSigmoid

• bool isNormalized

• bool isBatchAgnostic {true}

9.1.1

Detailed Description

The DetectionOutput plugin layer generates the detection output based on location and confidence predictions by

doing non maximum suppression. This plugin first decodes the bounding boxes based on the anchors generated. It

then performs non max suppression on the decoded bounding boxes. DetectionOutputParameters defines a set of

parameters for creating the DetectionOutput plugin layer. It contains:

100

Class Documentation

Parameters

shareLocation

If true, bounding box are shared among different classes.

varianceEncodedInTarget

If true, variance is encoded in target. Otherwise we need to adjust the predicted offset

accordingly.

backgroundLabelId

Background label ID. If there is no background class, set it as -1.

numClasses

Number of classes to be predicted.

topK

Number of boxes per image with top confidence scores that are fed into the NMS

algorithm.

keepTopK

Number of total bounding boxes to be kept per image after NMS step.

confidenceThreshold

Only consider detections whose confidences are larger than a threshold.

nmsThreshold

Threshold to be used in NMS.

codeType

Type of coding method for bbox.

inputOrder

Specifies the order of inputs {loc data, conf data, priorbox data}.

confSigmoid

Set to true to calculate sigmoid of confidence scores.

isNormalized

Set to true if bounding box data is normalized by the network.

isBatchAgnostic

Defaults to true. Set to false if prior boxes are unique per batch

9.1.2

Member Data Documentation

9.1.2.1

backgroundLabelId

int32 t nvinfer1::plugin::DetectionOutputParameters::backgroundLabelId

9.1.2.2

codeType

CodeTypeSSD nvinfer1::plugin::DetectionOutputParameters::codeType

9.1.2.3

confidenceThreshold

float nvinfer1::plugin::DetectionOutputParameters::confidenceThreshold

102

Class Documentation

9.1.2.11

shareLocation

bool nvinfer1::plugin::DetectionOutputParameters::shareLocation

9.1.2.12

topK

int32 t nvinfer1::plugin::DetectionOutputParameters::topK

9.1.2.13

varianceEncodedInTarget

bool nvinfer1::plugin::DetectionOutputParameters::varianceEncodedInTarget

The documentation for this struct was generated from the following file:

• NvInferPluginUtils.h

9.2

Dims Class Reference

Structure to define the dimensions of a tensor.

#include <NvInferRuntimeBase.h>

9.2.1

Detailed Description

Structure to define the dimensions of a tensor.

TensorRT can also return an invalid dims structure. This structure is represented by nbDims == -1 and d[i] == 0 for all

TensorRT can also return an ”unknown rank” dims structure. This structure is represented by nbDims == -1 and d[i]

== -1 for all d.

The documentation for this class was generated from the following file:

• NvInferRuntimeBase.h

9.3 nvinfer1::Dims2 Class Reference

103

9.3

nvinfer1::Dims2 Class Reference

Descriptor for two-dimensional data.

#include <NvInferLegacyDims.h>

Inheritance diagram for nvinfer1::Dims2:

nvinfer1::Dims32

nvinfer1::Dims2

nvinfer1::Dims3

nvinfer1::DimsHW

nvinfer1::Dims4

Public Member Functions

• Dims2 ()

Construct an empty Dims2 object.

• Dims2 (int32 t d0, int32 t d1)

Construct a Dims2 from 2 elements.

Additional Inherited Members

9.3.1

Detailed Description

Descriptor for two-dimensional data.

9.3.2

Constructor & Destructor Documentation

9.3.2.1

Dims2() [1/2]

nvinfer1::Dims2::Dims2 ( ) [inline]

Construct an empty Dims2 object.

9.3.2.2

Dims2() [2/2]

nvinfer1::Dims2::Dims2 (

int32 t d0,

int32 t d1 ) [inline]

Construct a Dims2 from 2 elements.

106

Class Documentation

Public Attributes

• int32 t nbDims

The rank (number of dimensions).

• int32 t d [MAX DIMS]

The extent of each dimension.

Static Public Attributes

• static constexpr int32 t MAX DIMS {8}

The maximum rank (number of dimensions) supported for a tensor.

9.5.1

Member Data Documentation

9.5.1.1

int32 t nvinfer1::Dims32::d[MAX DIMS]

The extent of each dimension.

9.5.1.2

MAX DIMS

constexpr int32 t nvinfer1::Dims32::MAX DIMS {8} [static], [constexpr]

The maximum rank (number of dimensions) supported for a tensor.

9.5.1.3

nbDims

int32 t nvinfer1::Dims32::nbDims

The rank (number of dimensions).

The documentation for this class was generated from the following file:

• NvInferRuntimeBase.h

NVIDIA TensorRT 8.6.11 API Reference for DRIVE OS (May 2023) - page 2