Dolphin (Document Image Parsing via Heterogeneous Anchor Prompting) is a novel multimodal document image parsing model following an analyze-then-parse paradigm. This repository contains the demo code and pre-trained models for Dolphin.

Document image parsing is challenging due to its complexly intertwined elements such as text paragraphs, figures, formulas, and tables. Dolphin addresses these challenges through a two-stage approach:

1. 🔍 Stage 1: Comprehensive page-level layout analysis by generating element sequence in natural reading order
2. 🧩 Stage 2: Efficient parallel parsing of document elements using heterogeneous anchors and task-specific prompts

Dolphin achieves promising performance across diverse page-level and element-level parsing tasks while ensuring superior efficiency through its lightweight architecture and parallel parsing mechanism.

Demo is coming soon within these days. Please keep tuned! 🔥

• 🔥 2025.05.20 The pretrained model and inference code of Dolphin are released.
• 🔥 2025.05.16 Our paper has been accepted by ACL 2025. Paper link: Arxiv.

1. Clone the repository:

   git clone https://github.com/ByteDance/Dolphin.git
   cd Dolphin

2. Install the dependencies:

   pip install -r requirements.txt

3. Download the pre-trained models using one of the following options:

   Option A: Original Model Format (config-based)

   Download from Baidu Yun or Google Drive and put them in the ./checkpoints folder.

   Option B: Hugging Face Model Format

   Visit our Huggingface model card, or download model by:

   # Download the model from Hugging Face Hub
   git lfs install
   git clone https://huggingface.co/ByteDance/Dolphin ./hf_model
   # Or use the Hugging Face CLI
   huggingface-cli download ByteDance/Dolphin --local-dir ./hf_model

Dolphin provides two inference frameworks with support for two parsing granularities:

• Page-level Parsing: Parse the entire document image into a structured JSON and Markdown format
• Element-level Parsing: Parse individual document elements (text, table, formula)

# Process a single document image
python demo_page.py --config ./config/Dolphin.yaml --input_path ./demo/page_imgs/page_1.jpeg --save_dir ./results

# Process all document images in a directory
python demo_page.py --config ./config/Dolphin.yaml --input_path ./demo/page_imgs --save_dir ./results

# Process with custom batch size for parallel element decoding
python demo_page.py --config ./config/Dolphin.yaml --input_path ./demo/page_imgs --save_dir ./results --max_batch_size 8

# Process a single document image
python demo_page_hf.py --model_path ./hf_model --input_path ./demo/page_imgs/page_1.jpeg --save_dir ./results

# Process all document images in a directory
python demo_page_hf.py --model_path ./hf_model --input_path ./demo/page_imgs --save_dir ./results

# Process with custom batch size for parallel element decoding
python demo_page_hf.py --model_path ./hf_model --input_path ./demo/page_imgs --save_dir ./results --max_batch_size 16

# Process a single table image
python demo_element.py --config ./config/Dolphin.yaml --input_path ./demo/element_imgs/table_1.jpeg --element_type table

# Process a single formula image
python demo_element.py --config ./config/Dolphin.yaml --input_path ./demo/element_imgs/line_formula.jpeg --element_type formula

# Process a single text paragraph image
python demo_element.py --config ./config/Dolphin.yaml --input_path ./demo/element_imgs/para_1.jpg --element_type text

# Process a single table image
python demo_element_hf.py --model_path ./hf_model --input_path ./demo/element_imgs/table_1.jpeg --element_type table

# Process a single formula image
python demo_element_hf.py --model_path ./hf_model --input_path ./demo/element_imgs/line_formula.jpeg --element_type formula

# Process a single text paragraph image
python demo_element_hf.py --model_path ./hf_model --input_path ./demo/element_imgs/para_1.jpg --element_type text

• 🔄 Two-stage analyze-then-parse approach based on a single VLM
• 📊 Promising performance on document parsing tasks
• 🔍 Natural reading order element sequence generation
• 🧩 Heterogeneous anchor prompting for different document elements
• ⏱️ Efficient parallel parsing mechanism
• 🤗 Support for Hugging Face Transformers for easier integration

We would like to acknowledge the following open-source projects that provided inspiration and reference for this work:

• Donut
• Nougat
• GOT
• MinerU
• Swin
• Hugging Face Transformers

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{dolphin2025,
  title={Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting},
  author={Feng, Hao and Wei, Shu and Fei, Xiang and Shi, Wei and Han, Yingdong and Liao, Lei and Lu, Jinghui and Wu, Binghong and Liu, Qi and Lin, Chunhui and Tang, Jingqun and Liu, Hao and Huang, Can},
  year={2025},
  booktitle={Proceedings of the 65rd Annual Meeting of the Association for Computational Linguistics (ACL)}
}