Back to Gap Map Model components / Dataset Processing Tools

img2dataset

Romain Beaumont
open source / Overall score: 2.4

img2dataset turns large lists of image URLs, with optional captions, into training-ready image datasets. It downloads, resizes and packages images into webdataset tar shards, parquet or TFRecord, and scales from one machine to Spark or Ray clusters. It skips images whose servers send noai-style X-Robots-Tag opt-out headers unless told otherwise, and was used to download the LAION image-text datasets.

Openness

5 high confidence
5.0
license
MIT (OSI)
source
public (rom1504/img2dataset)
core features withheld
no — an individual's project with no company, paid tier or hosted edition beside it

img2dataset is MIT-licensed and the whole tool is in its public repository. It is one developer's project, with nothing sold beside it.

Adoption

2 medium confidence
2.0

Adoption is the trailing month of PyPI downloads of img2dataset. A dataset build runs the tool on a few machines for days, so downloads count installs, not the size of the datasets built with it.

Capability

3 medium confidence
3.0

img2dataset does one job at very large scale: fetching and packaging billions of web images, as it did for the LAION datasets. Like CCNet, it is a single-approach pipeline, and quality filtering and deduplication are left to other tools.

  • https://raw.githubusercontent.com/rom1504/img2dataset/main/README.md recorded 2026-11-12

    The README says it "Can download, resize and package 100M urls in 20h on one machine", lists "laion400m 400M image/text pairs that can be downloaded in 3.5 days" and "laion5B 5B image/text pairs that can be downloaded in 7 days using 10 nodes", output formats webdataset, parquet and tfrecord, pyspark and ray distributors, and the default disallowed_header_directives "noai", "noimageai", "noindex" and "noimageindex".

Verified 2026-11-12