Agent infrastructure
pooled
Nehanth/pooled
Pooled runs large open models across browsers by splitting model layers among devices in a room over WebRTC and WebGPU. Users open a link to chat or use Code mode, and the room can expose an OpenAI/Anthropic-compatible API to other tools.
Topics
Related projects
mlx-serve2.4kNative LLM inference server for Apple Silicon with OpenAI- and Anthropic-compatible APIs, plus a macOS app offering chat, agent mode, multimodal interaction, and tool calling.
sglang38.7kA fast open-source inference framework for serving large language and multimodal models at scale.
web-llm19.4kA high-performance in-browser LLM inference engine with WebGPU acceleration and OpenAI-compatible APIs for building private web AI apps.
nano-vllm16.3kA lightweight vLLM reimplementation in about 1,200 lines of Python delivering fast, readable offline LLM inference.
LMCache12.3kA vendor-neutral KV cache management layer for LLM inference that reuses cached context to cut latency and boost throughput.
inference9.6kA unified model-serving library to deploy LLMs and multimodal models locally or in the cloud via OpenAI-compatible API.