Windy TranslateEvaluation method
Every score on this site comes from one screening run on 2026-08-04. Here is exactly what it measured and what it did not.
Translation pairs
- Test set: FLORES-200 dev split, the first 48 sentences per pair. FLORES is professionally translated and is not the corpus these models were trained on (OPUS / Tatoeba), so it does not reward memorisation.
- Metrics: chrF++ (character n-grams plus word bigrams, the WMT default) and BLEU, computed with sacrebleu.
- Bands, by chrF++: Excellent ≥ 60 (30 pairs), Good 45–60 (221), Usable 30–45 (284), Limited 15–30 (238), Not recommended < 15 (15). Median 36.79.
- Scored: 788 of 1,575 pair models.
Why 787 pairs have no score
| Reason | Models |
|---|---|
| FLORES-200 has no test set for the language (most of the long tail) | 733 |
| CTranslate2 (HPLT) models: measured, but that harness was found broken, so scores are withheld | 25 |
| Not part of the screening run | 25 |
| Same-language models (a score would only measure copying) | 4 |
The 8 directions Windy runs itself (full devtest)
Windy's own translation service uses 8 of these models (English to and from Spanish, French, German and Portuguese). Before switching to them on 2026-10-01 we measured each on all 1,012 sentences of the FLORES-200 devtest split against the model they replaced, NLLB-200-distilled-600M (whose weights are licensed non-commercial). The rule was: switch only if within 1.0 chrF++ of NLLB. All 8 passed.
| Direction | Model | chrF++ (ours) | NLLB-600M | Difference |
|---|---|---|---|---|
| en→es | translate-tc-big-en-es | 54.24 | 53.42 | +0.82 |
| es→en | translate-tc-big-cat_oci_spa-en | 57.69 | 57.46 | +0.23 |
| en→fr | translate-tc-big-en-fr | 70.57 | 68.26 | +2.31 |
| fr→en | translate-tc-big-fr-en | 67.53 | 66.01 | +1.52 |
| en→de | translate-en-de | 60.95 | 60.77 | +0.18 |
| de→en | translate-de-en | 64.75 | 65.25 | -0.50 |
| en→pt | translate-tc-big-en-pt | 69.95 | 68.56 | +1.39 |
| pt→en | translate-tc-bible-big-roa-en | 70.91 | 69.66 | +1.25 |
sacrebleu 2.6, chrF++ (word_order=2), one reference per sentence; differences under about 1 point are within noise, so the claim is "no worse than NLLB", not "better". These 1,012-sentence numbers are separate from the 48-sentence screening scores on the model pages.
Language verification
53 unscored models passed a GlotLID language-identification check on their output: the model writes the language it claims. That is evidence of identity, not of quality, so it is shown as a separate "language-verified" badge and never mixed into the bands.
Caveats
- 48 sentences is a screening sample. Small differences between pairs (a point or two of chrF++) are noise.
- FLORES-200 is news and encyclopedic prose. It says little about chat, menus, medicine or law.
- No human evaluation was performed.
- chrF++ is not "accuracy". A 60 does not mean 60% correct.
- Numbers for the three multilingual flagships come from their model cards (FLORES-200 devtest, 1,012 sentences, spBLEU/chrF) and were not re-measured here.
- Speech-to-text models are not benchmarked here yet.
What is next
Re-score every pair on the full FLORES-200 devtest split (1,012 sentences), fix the CTranslate2 harness and publish the HPLT scores, and add word-error-rate results for the speech models. Scores will be published either way, including the weak ones.