FlashInfer: Kernel Library for LLM Serving

A throughput-oriented high-performance serving framework for LLMs

Dynamic Memory Management for Serving LLMs without PagedAttention

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. Tensor…

Minimalist ML framework for Rust

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

A high-throughput and memory-efficient inference and serving engine for LLMs

Inference code for Llama models

SCQL (Secure Collaborative Query Language) is a system that allows multiple distrusting parties to run joint analysis without revealing their private data.

Running large language models on a single GPU for throughput-oriented scenarios.

Simple samples for TensorRT programming

Kernl lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackable.

🤗 Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.

Synthesizer for optimal collective communication algorithms

Repo for external large-scale work

Transformer related optimization, including BERT, GPT

Development repository for the Triton language and compiler

Microsoft Collective Communication Library

Large-scale model inference.

OneFlow is a deep learning framework designed to be user-friendly, scalable and efficient.

A baseline repository of Auto-Parallelism in Training Neural Networks

XGo is a programming language that reads like plain English. But it's also incredibly powerful — it lets you leverage assets from C/C++, Go, Python, and JavaScript/TypeScript, creating a unified so…

Kubernetes-native Deep Learning Framework

Training and serving large-scale neural networks with auto parallelization.

Flexible and powerful tensor operations for readable and reliable code (for pytorch, jax, TF and others)

PyTorch Implementation of OpenAI GPT-2

A PyTorch implementation of the Transformer model in "Attention is All You Need".