Introduction
When Vladimir Beskorovainyi compared two releases of Ru-Jev, an open-source Russian cross-encoder built on mmBERT-base, the initial results looked decisive: one version appeared more accurate, while two task categories showed steep regressions. None of those changes turned out to be real. A post-mortem revealed four distinct evaluation bugs that had distorted the numbers, including a training-validation leak and a benchmark that was only partially held out. This article walks through what happened, why it matters, and how teams can protect their own model comparisons from similar failures.
What Happened
Ru-Jev is a pointwise cross-encoder with 307 million parameters. It scores every (context, option) pair individually and applies a softmax across options to produce a distribution. The model runs on CPU and is released under the MIT license. Version 1.3 was designed to handle option names supplied by the caller in their own words, unlike version 1.1, which had memorized exact label phrasing from training data. To support this, v1.3 trained on multiple surface forms for each label, including transliterated English intent keys and clipped variants.
Bug One: A Shared Random Stream Moved the Validation Split
The first comparison table contained two unexplained regressions. Sentence acceptability dropped from 0.909 to 0.647, and a reasoning set fell from 0.939 to 0.596. The cause was the data builder’s use of random.shuffle without a dedicated generator. Because every random call in the file drew from the same module-level stream, adding label-writing styles to one task changed how many draws occurred before the next task’s validation slice was carved. Every subsequent task received a different split.
This created measurable leakage: roughly 90.3 percent of v1.3’s acceptability validation rows and 89.4 percent of its reasoning rows had already appeared in v1.1’s training data. The apparently strong 0.909 score was simply version 1.1 evaluated on rows it had seen during training. On genuinely held-out rows, that score was 0.650. The fix was to seed each split from the task name using random.Random(f"split:{task}"). As a control, the six tasks whose option sets never changed matched at 100 percent when rebuilt from the earlier commit, confirming the diagnosis.
Bug Two: Matching by Input Text Was Not Enough
After repairing the split, the author matched validation rows by input text and found nine comparable tasks. Because v1.3 could present option wordings that v1.1 had never encountered, scoring v1.1 on those rows measured robustness to renaming rather than true accuracy. Requiring both identical input text and identical option wording reduced the comparable set to six tasks.
A reported toxicity improvement from 0.722 to 0.954 had been presented as a capability gain, but it was an artifact of this mismatch. Version 1.1 actually scores 0.961 on toxicity when evaluated with the labels it trained on.
Bug Three: Two Denominators in One Table
The summary row of the comparison table—0.7948 versus 0.7920 across 5,078 decisions—was calculated from only one random half of the comparable rows. The per-task table beside it counted all 10,157 rows. A claimed calibration advantage of 0.0172 versus 0.0160 came from that same single split. Both figures appeared on the model card before being withdrawn.
Across all 10,157 rows, both releases answered exactly 8,120 correctly and disagreed on 1,680 rows, split evenly 840 to 840. Over 25 random splits, the calibration difference changed sign. The one surviving observation is narrower: v1.3 uses a lower fitted temperature in every split, making its raw scores less overconfident, but once each model receives its own temperature, the residual calibration error is identical.
Bug Four: An Integrity Check Against a Stale Reference
To estimate seed variance, the author retrained both recipes on a rented GPU and verified the remote data against a local copy using SHA-256 hashes, row counts, and per-task counts. Everything matched. The problem was that the local copy predated the commit that added transliteration styles, so the “augmented” seed runs had actually trained without augmentation. Their jargon accuracy fell within the baseline range and was initially misread as an unstable recipe.
Row counts could not detect the issue because the new styles changed option strings, not the number of rows. Counting distinct option strings per task did: 120 in the stale copy versus 198 after rebuilding. Those runs were excluded from all results.
Why This Matters
Evaluation errors can masquerade as model improvements or regressions, leading teams to publish conclusions that do not hold up. In this case, a release that only changed label wording appeared to cause large swings in accuracy because the validation data was leaking from training, labels were mismatched, and summary statistics were drawn from inconsistent subsets. For cross-encoders and other ranking-sensitive architectures, where the exact surface form of labels can influence scores, these methodological flaws are especially dangerous. Model cards, benchmark leaderboards, and production deployment decisions all depend on trustworthy evaluation.
Key Takeaways
- Seed every data split from the name of the entity being split, never from a random stream shared with other code.
- Define comparability by everything the model consumes, then compute and report the overlap between datasets.
- Measure leakage directly by checking what fraction of a new validation set appears in prior training data.
- Use a control that must not move; if a fix does not predict what stays the same, the diagnosis is incomplete.
- Keep one denominator per table and replace single-split figures with distributions across multiple random splits.
- Verify data artifacts by the content your change is supposed to alter, not by hashing against a copy of unknown age.
- Check every probe vocabulary against training data before calling it held out, and test wording at more than one candidate-set size.
Conclusion
Once the numbers were corrected, the author ran a broader wording test across ten label vocabularies on the same 1,000 routing questions, using both five and eighteen candidates per question. Two of the original six benchmark vocabularies turned out to be training styles for both releases, and a third was a training style only for the new release—meaning half the original benchmark was not genuinely held out. The effect size also depended heavily on the number of candidates, which had made one gain look smaller than it was.
On held-out word-based vocabularies, the new release performed better in both candidate settings, though not every gain survived statistical correction. No usable improvement appeared for abbreviations or opaque codes. The new model is also more confident overall, which increased high-confidence errors on some tasks. Because v1.3 differed from v1.1 in more than just label augmentation—including the split fix and a move from three to four training epochs—the comparison is not a clean ablation. The central lesson is that rigorous evaluation hygiene is as important as the model architecture itself.

Discussion
Join the conversation
Thoughtful reactions, questions, and follow-up ideas help shape the next story.