ConceptSeg-R1

ConceptSeg-R1: Segment Any Concept via Meta-Reinforcement Learning

Introduction • Get Started • Data • Checkpoints

• May 2026 — arXiv paper released 🎉

• 🌳 From Objects to Concepts
  We introduce a three-level concept hierarchy covering CI, CD, and CR concepts, pushing segmentation beyond category recognition.

• 🔁 From Instance Solving to Rule Induction
  Meta-GRPO enables the model to infer transferable task rules from visual demonstrations and apply them deductively to unseen queries.

• 🔗 Latent Concept Tokens for Frozen SAM 3
  We map MLLM reasoning states into implicit concept tokens in the SAM 3 prompt space, enabling reasoning-aware segmentation without fine-tuning SAM 3.

• ⚡ From Heavy Reasoning to Adaptive Inference
  The Shortcut Router dynamically balances SAM 3 efficiency and reasoning depth, enabling fast perception for simple cases and deeper reasoning for complex concepts.

conda create -n conceptseg-r1 python=3.10
conda activate conceptseg-r1
bash setup.sh

# Stage 1: SFT Training
bash run_grpo_multiimage_stage1.sh

# Stage 2: GRPO Training
# Note: Set `model_path` to the Stage 1 output checkpoint before running. （If you training encounter unexpected GPU OOM   despite sufficient VRAM,  try changing transformers_version to "4.49.0" in model_path/generation_config.json.）
bash run_grpo_multiimage_stage2.sh

bash eval_conceptseg.sh

bash eval_reasonseg.sh

# Test 4  cases
bash run_scripts/inference.sh

# Or test a specific case
python src/eval/inference_single_example.py \
    --model_path "path/to/model" \
    --infer_path "path/to/image" \
    --question "concept description" \
    --output_path "output/path"

all_meta.json is no longer tracked in this repository. Download all_meta.json.zip from GitHub Releases and run bash setup.sh to extract it before training.

Place datasets under a shared root directory (image_folders):

root/
├── isic2018/
├── rare/
├── Breast_Tumor/
├── transparent1024/
├── MGrounding-630k/
├── Polyp/
├── Shadow_detection/
├── MIG-Bench/
├── coco2014_Living/
├── CoSOD3k1024/
├── ultra_rare/
├── coco2014_Artifact/
├── fewshot1000/
├── DUTS/
├── ESDIDefects/
└── COD10K1024/

Evaluation uses the PySegMetric_EvalToolkit.

We reference the excellent open-source repos SAM 3, VLM-R1 and LENS. Thanks to their authors for the valuable contributions to the community.

If you find this work useful, please consider starring ⭐ and citing the repo!

@misc{zhao2026conceptseg,
      title={ConceptSeg-R1: Segment Any Concept via Meta-Reinforcement Learning}, 
      author={Yuan Zhao and Youwei Pang and Jiaming Zuo and Wei Ji and Kailai Zhou and Bin Fan and Yunkang Cao and Lihe Zhang and Xiaofeng Liu and Huchuan Lu and Weisi Lin and Dacheng Tao and Xiaoqi Zhao},
      year={2026},
      eprint={2605.20385},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.20385}, 
}