Back to Gap Map Model components / Dataset Processing Tools

Data-Juicer

open source / Overall score: 2.2

Data processing system for foundation models: 200+ composable operators for cleaning, deduplicating, synthesizing, and analyzing text and multimodal LLM training data.

Alibaba Data-Juicer (datajuicer/data-juicer, formerly modelscope); PyPI package py-data-juicer.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public
core features withheld: no

Fully OSI-licensed (Apache-2.0), full source public.

https://github.com/datajuicer/data-juicer/blob/main/LICENSE recorded 2026-08-13
Apache License Version 2.0 text
https://raw.githubusercontent.com/datajuicer/data-juicer/main/LICENSE recorded 2026-09-29
Apache License text: "Apache License" and "Version 2.0, January 2004".
https://raw.githubusercontent.com/datajuicer/data-juicer/main/pyproject.toml recorded 2026-09-29
pyproject.toml declares license = {text = "Apache-2.0"} and classifier "License :: OSI Approved :: Apache Software License".
https://raw.githubusercontent.com/datajuicer/data-juicer/main/README.md recorded 2026-09-29
README License section: "Data-Juicer is released under the [Apache License 2.0](LICENSE)." Searched the body for enterprise, paid, pricing and license key: none.

Adoption

1 high confidence

1.0

PyPI downloads of the declared py-data-juicer package are the only usage figure published for this tool.

https://pypistats.org/api/packages/py-data-juicer/recent recorded 2026-08-13
2,962 downloads in the trailing 30 days for py-data-juicer

Capability

4 medium confidence

4.0

The operator library spans text, image, audio and video as well as mixed multimodal data.

https://raw.githubusercontent.com/datajuicer/data-juicer/main/README.md recorded 2026-09-29
README states "200+ operators** spanning text, image, audio, video, and multimodal data" and lists deduplication, cleaning, synthesis and analysis capabilities such as "Deduplicate 5TB in 2.8h using 1280 cores".

Verified 2026-09-24