Agent infrastructure
airllm
lyogavin/airllm
A Python inference engine that streams huge language models layer by layer to run 70B to 671B models on small GPUs.
Topics
Related projects
Chinese-LLaMA-Alpaca18.9kChinese-extended LLaMA and instruction-tuned Alpaca large language models, releasing LoRA weights plus training and local quantized deployment guides.
ms-swift16.1kA full-pipeline framework for fine-tuning, aligning, evaluating, and deploying hundreds of LLMs and multimodal models.
LocalAI50.3kAn open-source engine to run LLMs, vision, and audio models locally with OpenAI-compatible APIs.
sglang38.7kA fast open-source inference framework for serving large language and multimodal models at scale.
OpenLLM12.6kAn open-source platform for self-hosting LLMs and serving them as OpenAI-compatible APIs in the cloud.
lmdeploy8.1kA toolkit for compressing, deploying, and serving LLMs with fast inference engines, quantization, and distributed serving.