We introduce LDA-1B, a robot foundation model that scales through universal embodied data ingestion by jointly learning dynamics, policy, and visual forecasting, assigning distinct roles to data of varying quality.

• ✨ Key Features
• 🛠 Environment Setup
• 🧩 Model Architecture
• 💡 Training & Evaluation
• 🙏 Acknowledgements
• ✍️ Citation

• [2026-02-12] We publish LDA-1B, check our paper here.

git clone https://github.com/jiangranlv/latent-dynamics-action.git LDA
cd LDA

Create and activate a conda environment with the required dependencies, for example:

# Create a conda environment
conda create -n LDA python=3.10
conda activate LDA

# Install requirements
pip install -r requirements.txt

# Install FlashAttention2 with a version compatible with your PyTorch and CUDA versions
pip install flash-attn --no-build-isolation

# Install LDA
pip install -e .

Follow the instruction in Qwen3-VL and DINOv3 to download the pretrained VLM and vision encoder.

or you could directly download the pretrained from the following link:

• Qwen3-VL-4B: link🤗

• DINO-ViT-S: link🤗

LDA jointly denoises action chunks and future visual latent under multiple co-training objectives. Conditioned on VLM tokens, diffusion timesteps, and task embeddings, the model adopts a multimodal diffusion transformer architecture.

Core components:

• Language and Vision Encoder: Qwen3-VL (4B) → extracts semantics information

• Latent Visual Representation: DINOv3-ViT-S → extracts spatial features (frozen during training)

• MM-DiT Backbone: A 16-layer multi-modal diffusion transformer (hidden_dim=1536, num_heads=32).

Below is a description of the MM-DiT forward pass.

We provide training and evaluation scripts for the RoboCasa-GR1 dataset. Follow the steps described in Robocasa_tabletop to reproduce our results.

We also provide a demo dataset for quick debugging and validation.

You can launch training by running this script.

Make sure to update the following arguments in the script before execution:

• base_vlm: local path to the Qwen3 checkpoint
• vision_encoder_path: local path to the DINOv3 checkpoint
• data_root_dir: dataset root directory
• data_mix: target dataset name, defined in data_config.py
• run_root_dir: directory for saving checkpoints
• run_id: name used for the current training run

In addition to closed-loop evaluation in simulation (interactive execution with environment feedback), we also provide an open-loop evaluation interface for offline assessment. Open-loop evaluation quantitatively measures model performance by comparing predicted action sequences against ground-truth demonstrations from the dataset, without environment interaction.

bash LDA/scripts/eval_scripts/eval_lerobot_datasets_LDA.sh

The following features are planned for future implementation:

• Pre-trained model checkpoints.
• Pre-training data.
• Data preprocess scripts.

Our code is built upon starVLA and mmdit. These code serve as an essential foundation for our implementation, and we deeply appreciate the time, effort, and expertise they shared with the community.

If you find our work useful, please cite us:

@article{lyu2026lda,
  title={LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestion},
  author={Lyu, Jiangran and Liu, Kai and Zhang, Xuheng and Liao, Haoran and Feng, Yusen and Zhu, Wenxuan and Shen, Tingrui and Chen, Jiayi and Zhang, Jiazhao and Dong, Yifei and others},
  journal={arXiv preprint arXiv:2602.12215},
  year={2026}
}

This work and the dataset are licensed under CC BY-NC 4.0.