Libraries & frameworks
TensorRT
NVIDIA/TensorRT
NVIDIA's SDK for high-performance deep learning inference on GPUs, with plugins, parsers, and samples for optimized deployment.
Topics
Related projects
faster-whisper26.7kA fast Whisper transcription library built on CTranslate2, with quantization, batching, VAD filtering, and word timestamps.
ncnn24kA high-performance neural network inference framework optimized for mobile, embedded, and desktop model deployment.
tfjs19.2kA hardware-accelerated JavaScript machine learning library for browser and Node.js, offering tensor ops, Layers APIs, and converters for training and deployment.
openvino11.1kAn open-source toolkit for optimizing and deploying deep learning inference across CPUs, GPUs, and AI accelerators.
cutlass10.7kCUDA C++ template library and Python DSLs for high-performance GEMM and linear algebra with reusable tiling and Tensor Core abstractions.
nano-vllm16.3kA lightweight vLLM reimplementation in about 1,200 lines of Python delivering fast, readable offline LLM inference.