Agent infrastructure
krasis
brontoguana/krasis
A hybrid LLM inference runtime for running very large models on consumer GPUs with limited VRAM. Uses GPU prefill, decoding, and hot-cold expert management to enable local execution for individual practitioners.
Topics
Related projects
kimi-k3-in-c10.2kA portable C99 inference engine that runs the 2.78T-parameter Kimi K3 model on a single CPU with streaming.
mlx-dspark736A native MLX speculative-decoding inference engine that speeds up local LLMs on Apple Silicon.
litgpt13.8kA developer framework with from-scratch implementations of 20+ LLMs plus proven recipes to pretrain, finetune, evaluate, and deploy them.
lmdeploy8.1kA toolkit for compressing, deploying, and serving LLMs with fast inference engines, quantization, and distributed serving.
plano7.1kAn AI-native proxy and data plane for agentic apps offering LLM routing, agent orchestration, guardrails, and observability.
helicone6.3kAn open-source AI gateway and LLM observability platform with unified routing, tracing, and cost analytics.