Released September 2024, MMMLU translates the MMLU question set into multiple languages and reports an aggregate accuracy across them. Because the number and choice of languages included varies between evaluation runs, two model cards both citing an 'MMMLU' score are only comparable if they cover the same language set.