NVIDIA Tensorrt is an ecosystem of tools for developers to achieve high-performance deep learning inference. TensorRT includes inference compilers, runtimes, and model optimizations that deliver low latency and high throughput for production applications. The TensorRT ecosystem includes the TensorRT compiler, TensorRT-LLM, TensorRT Model Optimizer, and TensorRT Cloud.

NVIDIA Tensorrt Ai Features

• NVIDIA Tensorrt LLM is an open-source library that accelerates and optimizes inference performance of large language models on the NVIDIA AI platform with a simplified Python API.

• Developers accelerate LLM performance on NVIDIA GPUs in the data center or on workstation GPUs, including NVIDIA RTX systems on native Windows, with the same seamless workflow.

• NVIDIA Tensorrt Cloud is a developer focused service for generating hyper-optimized engines for given constraints and KPIs. Given an LLM and inference throughput/latency requirements, a developer can invoke Tensorrt Cloud service using a command-line interface to hyper-optimize a TensorRT-LLM engine for a target GPU.

• NVIDIA Tensorrt Model Optimizer is a unified library of state-of-the-art model optimization techniques, including quantization, sparsity, and distillation.