Irish

The first rigorous benchmark evaluating LLM quality beyond English, across all 24 official EU languages in professional and institutional contexts.

Methodology summary: Each of the 24 EU languages is represented by at least 120 native-authored tasks. Tasks span four genres: legislative text comprehension, administrative correspondence drafting, professional document summarization, and cross-lingual knowledge retrieval. Scoring uses a combination of automated metrics (BLEU, BERTScore) and human evaluation for a representative 15% sample per language.