A lightweight CUDA inference engine for Qwen3 models, designed for consumer multi-GPU setups (for example, dual RTX 3080 Ti).

As of March 12, 2026, Ember is focused on a narrow goal: high-performance Qwen inference on consumer NVIDIA GPUs.

Current:

• Stable Qwen3 dense CUDA inference
• Native safetensors loading
• Multi-GPU pipeline-parallel runtime
• Minimal CLI and server path

Next:

• Qwen3.5 hybrid architecture support
• DeltaNet + Gated Attention runtime
• HF-aligned correctness and regression harness

Then:

• Qwen3.5 35B-A3B MoE inference
• Dual-GPU + CPU offload execution path
• Reproducible benchmarks and OpenAI-compatible serving

1. Build (example: RTX 3080 Ti, SM=86):

cmake -S . -B build -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel

By default, Ember builds only the core inference target (ember). To also build optional tests/benchmarks:

cmake -S . -B build \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DEMBER_BUILD_TESTS=ON \
  -DEMBER_BUILD_BENCHMARKS=ON

2. Download a model (safetensors format; first run may require huggingface-cli login):

# If huggingface-cli is missing: pip install -U huggingface_hub
huggingface-cli download Qwen/Qwen3-0.6B --local-dir ./qwen3-0.6b

3. Run:

./build/ember -m ./qwen3-0.6b -p "Hello, my name is"

English (default):

• Contributing
• Development Guide
• Testing and Regression
• Architecture Overview
• Sampler Deep Dive
• Benchmark Handbook

Chinese:

• README (Chinese)
• Sampler Deep Dive (Chinese)
• Benchmark Handbook (Chinese)

• Native CUDA implementation: full control over compute flow, without ggml/llama.cpp.
• Direct safetensors loading: loads HuggingFace format natively, no conversion step.
• Pipeline parallelism: multi-GPU layer split with memory-aware allocation.
• FP16 compute: FP16 weights/activations, GEMM via cuBLAS.
• Custom kernels: RMSNorm, RoPE, Softmax, SiLU, Attention, and more.

ember/
|-- apps/ember_cli/         # CLI entry point (main.cpp)
|-- cli/                    # CLI argument parsing
|-- core/                   # Core abstractions (pure C++, no CUDA dependency)
|-- runtime/                # Scheduling and device mapping runtime logic
|-- formats/                # safetensors/config loaders
|-- backends/cuda/          # CUDA runtime + kernels
|-- benchmarks/             # Performance and interconnect benchmarks
|-- tests/                  # Unit tests and smoke tests
|-- scripts/                # CI and alignment scripts
`-- docs/                   # Current design and testing docs

• CMake 3.18+
• CUDA Toolkit 11.0+ (12.x recommended)
• C++17 compiler

cmake -S . -B build -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_BUILD_TYPE=Release  # RTX 3080 Ti
cmake --build build --parallel

Common CUDA architecture values:

• 86 - RTX 3080/3090
• 89 - RTX 4090
• 80 - A100

# Single-GPU inference
./build/ember -m /path/to/qwen3-4b -p "Hello, my name is"

# Dual-GPU inference
./build/ember -m /path/to/qwen3-14b --devices 0,1 -p "Explain quantum computing"

# Interactive mode
./build/ember -m /path/to/qwen3-4b -i

• Qwen3-0.6B
• Qwen3-1.7B
• Qwen3-4B
• Qwen3-8B
• Qwen3-14B (typically needs dual GPUs)

Models should be downloaded from HuggingFace in safetensors format:

# If huggingface-cli is missing: pip install -U huggingface_hub
huggingface-cli download Qwen/Qwen3-4B --local-dir ./qwen3-4b

1. Tensor: lightweight view (shape + dtype + data ptr), no ownership.
2. Session: inference session state including KV cache.
3. IRuntime: backend interface (CUDA now, extensible in future).
4. DeviceMap: layer-to-device mapping for pipeline parallelism.

Input IDs
    |
    v
Embedding Lookup (GPU 0)
    |
    v
+-----------------------------+
| Layer 0-N (may span GPUs)   |
|   |- Input LayerNorm        |
|   |- QKV Projection         |
|   |- RoPE                   |
|   |- KV Cache Update        |
|   |- Attention (Q@K^T -> V) |
|   |- O Projection           |
|   |- Residual Add           |
|   |- Post-Attn LayerNorm    |
|   |- MLP (SwiGLU)           |
|   `- Residual Add           |
+-----------------------------+
    |
    v
Final LayerNorm
    |
    v
LM Head -> Logits
    |
    v
Sampling -> Next Token

Ember uses layer-wise pipeline parallelism. Example split:

GPU 0: Embedding + Layers 0-13
GPU 1: Layers 14-27 + LM Head

Hidden states are transferred with cudaMemcpyPeer.

Actual throughput depends on hardware and model size.

• M0: Single-GPU inference baseline
• M1: Dual-GPU pipeline parallelism
• M2: Quantization support (INT8/INT4)
• M3: FlashAttention optimization

If you use Ember in a paper or report, cite the Zenodo archive:

• DOI: https://doi.org/10.5281/zenodo.18477269

Apache-2.0

• llama.cpp - architecture inspiration
• HuggingFace Transformers - model format ecosystem
• Qwen - model weights