img2dataset
Romain Beaumontimg2dataset turns large lists of image URLs, with optional captions, into training-ready image datasets. It downloads, resizes and packages images into webdataset tar shards, parquet or TFRecord, and scales from one machine to Spark or Ray clusters. It skips images whose servers send noai-style X-Robots-Tag opt-out headers unless told otherwise, and was used to download the LAION image-text datasets.
Openness
5 high confidence- license
- MIT (OSI)
- source
- public (rom1504/img2dataset)
- core features withheld
- no — an individual's project with no company, paid tier or hosted edition beside it
img2dataset is MIT-licensed and the whole tool is in its public repository. It is one developer's project, with nothing sold beside it.
- https://pypi.org/pypi/img2dataset/json recorded 2026-11-12
PyPI lists img2dataset 1.47.0, uploaded 2025-08-09, with the license field "MIT".
- https://raw.githubusercontent.com/rom1504/img2dataset/main/LICENSE recorded 2026-11-12
The LICENSE is the standard MIT License text, "Copyright (c) 2021 Romain Beaumont".
- https://raw.githubusercontent.com/rom1504/img2dataset/main/README.md recorded 2026-11-12
The README installs with "pip install img2dataset" and documents the CLI, output formats and distributed modes from the repository. Searched the body for enterprise, pricing, paid and license key: none.
Adoption
2 medium confidenceAdoption is the trailing month of PyPI downloads of img2dataset. A dataset build runs the tool on a few machines for days, so downloads count installs, not the size of the datasets built with it.
- https://pypistats.org/api/packages/img2dataset/recent recorded 2026-11-12
last_month 22,357, last_week 9,050, last_day 991 for the img2dataset package.
Capability
3 medium confidenceimg2dataset does one job at very large scale: fetching and packaging billions of web images, as it did for the LAION datasets. Like CCNet, it is a single-approach pipeline, and quality filtering and deduplication are left to other tools.
- https://raw.githubusercontent.com/rom1504/img2dataset/main/README.md recorded 2026-11-12
The README says it "Can download, resize and package 100M urls in 20h on one machine", lists "laion400m 400M image/text pairs that can be downloaded in 3.5 days" and "laion5B 5B image/text pairs that can be downloaded in 7 days using 10 nodes", output formats webdataset, parquet and tfrecord, pyspark and ray distributors, and the default disallowed_header_directives "noai", "noimageai", "noindex" and "noimageindex".
Verified 2026-11-12