Agent infrastructure
server
triton-inference-server/server
NVIDIA's open-source inference serving software for deploying AI models efficiently across cloud and edge.
Topics
Related projects
skypilot10.8kA system to run and scale AI jobs on any cloud or cluster, with one interface for GPUs, scheduling, and serving.
onnxruntime22.3kA cross-platform high-performance inference and training accelerator for ML models with hardware acceleration and graph optimizations.
freetoken16.8kFreeToken is an edge-native Mixture-of-Experts inference engine that runs large open-weight models on consumer hardware, with a desktop app, CLI, and OpenAI/Anthropic-compatible APIs.
nano-vllm16.3kA lightweight vLLM reimplementation in about 1,200 lines of Python delivering fast, readable offline LLM inference.
wandb11.3kAn experiment tracking and observability platform for logging, visualizing, and managing machine learning training runs.
inference9.6kA unified model-serving library to deploy LLMs and multimodal models locally or in the cloud via OpenAI-compatible API.