Frontier AI models score worse than a coin flip on grammar in some languages, according to TrainAI
New linguist-designed benchmark, M-GATE, gives enterprises an independent way to compare how leading AI models perform across 30 languages, helping them choose the right model for every market
MAIDENHEAD, England, Aug. 25, 2026 /PRNewswire/ -- RWS (AIM: RWS.L), a global AI solutions company, today launched M-GATE (Multilingual Grammar, Accuracy in Translation & Efficiency), an independent benchmark that measures whether frontier AI models genuinely understand the languages they claim to support. Developed by RWS's TrainAI data services team, M-GATE evaluates 70 models on grammar, translation accuracy and tokenizer efficiency across 30 languages – from widely spoken ones to under-served languages like Kinyarwanda, Basque and Fijian – providing a reliable, independent signal for multilingual model selection.
What the benchmark reveals
No single model wins across the board. M-GATE tests grammar using a binary test – a sentence is either correct or incorrect – so random guessing scores around 50%. Yet results diverge sharply even within the same language. For example, on Fijian grammar, xAI's Grok 4.20 tops the entire field, while OpenAI's GPT-5.5, the benchmark's best translator, falls below random chance, and Meta's Muse Spark scores just 23%, near the bottom of all 70 models tested – blind spots that popular reasoning benchmarks never surface.
Google's Gemini 3.1 Pro Preview currently tops the grammar leaderboard overall across 70 models and 30 languages, while OpenAI's GPT-5.5 leads on roundtrip translation, which takes a sentence from English into the target language and back – but strength in one language rarely carries to the next.
Four patterns stood out:
- Flagship status doesn't predict grammar performance. M-GATE's "stumper" sentences, designed to probe tricky language-specific grammatical features, humble nearly every model, with many – including some top-tier models – scoring at or below random guessing in some languages. One example stumper sentence, "Everything I told you is what I thought I had said I would," sounds awkward enough to be wrong, but it isn't. That's exactly the kind of distinction the models are being asked to make on M-GATE.
- Tokenizer costs vary sharply by language. The heaviest tokenizers use more than ten times as many tokens to process some languages, such as Khmer, as they do English. Claude's tokenizer usage looks balanced overall. But in absolute terms, it packs the fewest characters per token of any major lab in the benchmark – which drives costs up.
- Speed varies enormously. The slowest models average roughly 100 times the latency of the fastest, with worst-case responses measured in minutes.
- The gap is closing where it matters most. Translation quality on low-resource languages has roughly doubled across the models benchmarked – progress that reasoning benchmarks don't measure – but meaningful gaps remain in the hardest languages.
"Enterprise teams are being asked to choose between models on the strength of a language support list – '30 languages,' '50 languages' – with no way to check whether any of the claims hold up," said Vasagi Kothandapani, CEO of TrainAI by RWS. "That's an expensive way to make a decision. M-GATE gives them a straight answer for each language that matters to their business, so they can choose the right model with confidence instead of making an expensive mistake."
How M-GATE works
M-GATE is an automated, linguist-designed benchmark that evaluates models on three dimensions:
- Grammar proficiency: 100 linguist-crafted "stumper" sentences per language – half containing deliberate errors, half correct but tricky – designed to probe tricky grammar rules specific to each language.
- Roundtrip translation accuracy: 100 source sentences spanning seven challenge categories – discourse pragmatics, non-compositional, implicit content, structural complexity, lexical disambiguation, referential precision (and control) – to test whether meaning survives a translate-and-back journey.
- Tokenizer efficiency: a measure of how many tokens each model consumes to process similar text, and how quickly it responds – both tied directly to cost and speed.
Tomáš Burkert, Head of Innovation at TrainAI by RWS, added, "The uncomfortable truth is that some of the most capable models on the market perform worse than random guessing in certain languages – and the benchmarks everyone quotes can never tell you that. We built M-GATE to measure the thing every buyer assumes and no one checks – whether a model actually holds up in the language you're about to ship it in."
Why this matters for enterprises
The findings come as businesses rapidly expand their use of AI-generated and AI-translated content across global markets. RWS's Content Unlocked 2026 study of 200 senior enterprise content leaders found that while 86% said AI had accelerated content creation, 65% said it had simultaneously slowed localization through additional rework. And most multilingual benchmarks test reasoning, math or knowledge expressed in a language, not command of the language itself.
M-GATE gives organizations another way to interrogate model performance before deployment – comparing model performance at the individual language level rather than having to assume a strong overall benchmark score will translate into equally strong multilingual performance.
RWS does not develop general-purpose frontier AI models, enabling its TrainAI practice to benchmark competing systems independently. Visit M-GATE to explore the benchmark and see how today's top models really perform across languages.
Notes to editors
- More than 70 AI models from leading companies have been evaluated on M-GATE, including Alibaba, Anthropic, ByteDance, DeepSeek, Google, Meta, MiniMax, Mistral AI, Moonshot AI, NVIDIA, OpenAI, Thinking Machines, SpaceXAI and Z.ai. Cohere has been purposely excluded from the benchmark as M-GATE aims to maintain independence (RWS and Cohere recently partnered to develop Language Weaver Pro, the world's most advanced AI translation platform).
- Every model is scored automatically. Grammar is graded by exact match against linguist-established yes or no answers, while roundtrip translation is scored by a three-provider LLM judge panel validated by human linguists to avoid single-provider bias.
About us
RWS is a global AI solutions company empowering the world's most trusted enterprise AI.
Our proprietary Cultural Intelligence Layer, powered by 250,000+ data specialists, cultural and language experts and deep domain professionals, backed by 47+ patents, makes enterprise AI culturally fluent, contextually accurate and secure, ensuring every interaction reflects a brand's tone, context and customer values.
Through our Generate, Transform and Protect segments, we deliver intelligent content, enterprise knowledge, large-scale localization and IP protection for global growth. Trusted by 80+ of the world's top 100 brands, RWS provides the confidence, governance and expertise organizations need to deploy AI safely, responsibly and at scale.
Headquartered in the UK, RWS is listed on AIM.
More information: rws.com.
View original content to download multimedia:https://www.prnewswire.com/news-releases/frontier-ai-models-score-worse-than-a-coin-flip-on-grammar-in-some-languages-according-to-trainai-302859350.html
SOURCE RWS
