Agent infrastructure
LMCache
LMCache/LMCache
A vendor-neutral KV cache management layer for LLM inference that reuses cached context to cut latency and boost throughput.
Topics
Related projects
sglang38.7kA fast open-source inference framework for serving large language and multimodal models at scale.
nano-vllm16.3kA lightweight vLLM reimplementation in about 1,200 lines of Python delivering fast, readable offline LLM inference.
inference9.6kA unified model-serving library to deploy LLMs and multimodal models locally or in the cloud via OpenAI-compatible API.
beellama.cpp1.2kllama.cpp fork focused on local LLM inference efficiency, using KVarN, KV-cache precision tailoring, and low-bit quantization to support longer context with better precision in the same VRAM.
TensorRT-LLM15kNVIDIA's open-source library for optimizing LLM inference with custom kernels and a high-performance runtime.
vllm-omni7.5kEfficient inference-serving framework for omni-modal models, unifying text, image, audio, video, and action inputs under one serving architecture for developers deploying multimodal applications.