Back to Gap Map Model components / Language-specific evaluation sets

AmericasNLP shared-task data

AmericasNLP
restricted / Overall score: 1.0

The AmericasNLP shared-task data pairs Spanish with ten Indigenous languages of the Americas, including Aymara, Bribri, Asháninka, Guarani, Wixarika, Nahuatl, Otomí, Quechua, Shipibo-Konibo and Rarámuri, for machine translation. Training sets gather existing parallel corpora, while development and test sets were translated by hand for the task. The AmericasNLP workshop organizers publish it with each shared task.

Openness

2 medium confidence
2.0
license
not-clearly-stated-on-card (No LICENSE file in americasnlp2021 (LICENSE, LICENSE.md, LICENSE.txt all 404); README only asks users to cite each source corpus.)
access
public
dataset_card
partial (README lists source corpora per language but does not describe the dev/test composition; the shared-task paper does.)

The files sit in public GitHub repositories, but no license is stated, and the training portions come from corpora that each keep their own terms.

Adoption

1 low confidence
1.0

GitHub stars summed across the shared-task repositories, the only public channel for the data; a star is attention rather than use.

Capability

1 medium confidence
1.0

In the typical language AmericasNLP shared-task data targets, its deepest test has four hundred to sixteen hundred items, but in 9 of its 10 languages that test comes from another dataset with nothing independent in the language to back it. Counting only what can be corroborated, the typical language has fewer than a hundred. An item count says how much a set can test, not how precisely it ranks models.

  • https://aclanthology.org/2021.americasnlp-1.23.pdf recorded 2026-11-08

    Table 1 gives Train, Dev and Test sizes per language (Asháninka cni 3883 883 1002; Aymara aym 6531 996 1003; Bribri bzd 7508 996 1003; Guarani gn 26032 995 1003; Nahuatl nah 16145 672 996; Otomí oto 4889 599 1001; Quechua quy 125008 996 1003; Rarámuri tar 14721 995 1002; Shipibo-Konibo shp 14592 996 1002; Wixarika hch 8966 994 1003); dev and test are "a manual translation of the Spanish version of the multilingual XNLI dataset"; "we leveraged individual pairs of parallel sentences from AmericasNLI"; "Systems in Track 1 were allowed to use the development set as part of the training data"; for Otomí "the development and test sets are from the Ñûhmû de Ixtenco, Tlaxcala, variant (otz)" while "most texts belong to the Valle del Mezquital dialect (ote)" in training; Nahuatl's dev and test translations "belong to Nahuatl Central/Nahuatl de la Huasteca (Hidalgo y San Luis Potosí) dialects"

  • https://aclanthology.org/2021.americasnlp-1.23/ recorded 2026-11-08

    "participants submitted machine translation systems for up to 10 indigenous languages. Overall, 8 teams participated with a total of 214 submissions."

  • https://aclanthology.org/2023.americasnlp-1.23.pdf recorded 2026-11-08

    Table 1 repeats the 2021 sizes and adds "Chatino ctp Oto-Manguean 357 499 1000"; "For this edition of the shared task, we use the same evaluation set and additionally introduce a new evaluation dataset, created from Mexican court proceedings, for Spanish–Chatino"; "Translations are jointly done by two professional translators, who are native San Juan Quiahije Chatino speakers"; "outside of the development set and any prohibited datasets, teams are allowed to collect and train on an unlimited amount of external data"

  • https://raw.githubusercontent.com/AmericasNLP/americasnlp2023/main/README.md recorded 2026-11-08

    Baseline chrF table for aym, bzd, cni, gn, hch, nah, oto, quy, shp, tar; baseline is the Helsinki-NLP 2021 system.

Verified 2026-09-24