Model components / Dataset Processing Tools

Data-Juicer

Data processing system for foundation models: 200+ composable operators for cleaning, deduplicating, synthesizing, and analyzing text and multimodal LLM training data.

Alibaba Data-Juicer (datajuicer/data-juicer, formerly modelscope); Apache-2.0, very active (v1.5.3, Jun 2026). PyPI package py-data-juicer. 200+ operators for multimodal LLM data processing.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public
core-gated: ungated

Fully OSI-licensed (Apache-2.0), full source public.

https://github.com/datajuicer/data-juicer/blob/main/LICENSE recorded 2026-07-21
Apache License Version 2.0 text

Adoption

2 high confidence

2.0

~3.1k PyPI downloads/month (py-data-juicer) and ~6.7k GitHub stars; actively developed.

https://pypistats.org/packages/py-data-juicer recorded 2026-07-21
Downloads last month: ~3,107

Capability

4 medium confidence

4.0

Broadest operator coverage among the open tools; multimodal. Benchmark participation but no single frontier-topping corpus of its own.

https://github.com/datajuicer/data-juicer recorded 2026-07-21
200+ operators for LLM data processing; active

Unchanged since 2026-07-30 (last edited, not re-checked)