TensorRT-LLM

Peak throughput on NVIDIA GPUs

Inference serverInference engine (NVIDIA)Apache-2.0Runs locallyC++/PythonAdvanced

OSAI Pulse ⓘ ★★★★★★★★★★ /100 signals tracked

🐳 Docker pulls 📦 PyPI downloads / month 📦 npm downloads / month 🚀 Latest release ·

Visit website →View on GitHub

What is TensorRT-LLM?

TensorRT-LLM compiles models into highly optimized NVIDIA kernels with in-flight batching, quantization and multi-GPU tensor parallelism — the reference for squeezing maximum tokens per second from NVIDIA hardware.

Why people choose TensorRT-LLM

Best-in-class throughput on NVIDIA hardware
FP8/INT4 quantization with official support
Deep integration with Triton and NVIDIA stack

TensorRT-LLM at a glance

Category	Inference server
Type	Inference engine (NVIDIA)
License	Apache-2.0
Runs locally	Yes
Built with	C++/Python
Skill level	Advanced
Best for	maximum performance on NVIDIA data-center GPUs

Open-source alternatives to TensorRT-LLM

Other open-source inference server tools worth comparing:

vLLMHigh-throughput serving for production

TGIHugging Face's production text server

SGLangFast serving with structured outputs

LMDeployToolkit for compressing and serving LLMs

Aphrodite EngineHigh-throughput LLM serving

OpenLLMServe any open model as an OpenAI API in one command

KTransformersRun huge MoE models on one consumer GPU

Ray ServeScale model serving across a cluster

BentoMLPackage any model into a production API

TensorRT-LLM head-to-head

TensorRT-LLM vs vLLM TensorRT-LLM vs TGI TensorRT-LLM vs SGLang TensorRT-LLM vs LMDeploy TensorRT-LLM vs Aphrodite Engine TensorRT-LLM vs OpenLLM TensorRT-LLM vs KTransformers TensorRT-LLM vs Ray Serve TensorRT-LLM vs BentoML

FAQ

Is TensorRT-LLM free?

TensorRT-LLM is free and open-source (Apache-2.0 license), so you can use, self-host and modify it at no cost.

Can I run TensorRT-LLM locally?

Yes. TensorRT-LLM is designed to run on your own machine or server, keeping your data private.

What is the best alternative to TensorRT-LLM?

Popular open-source alternatives include vLLM, TGI, SGLang. See the comparisons above to choose.

Explore more open-source AI

Browse the full directory of open-source AI tools, models and projects — updated daily.

Browse all tools →