Methodology
What We Measure
ModelRanking benchmarks 26 LLMs from 8 providers (OpenAI, Anthropic, Google, Mistral, DeepSeek, Groq, xAI, Cohere) on 129 distinct task families. Task families span document extraction, classification, code generation, math and reasoning, translation, summarization, image extraction, and video understanding.
Each task family has a generator that produces synthetic test workloads and a probe that evaluates outputs. Generators produce deterministic input/output pairs — no human labeling, no crowd-sourcing, no ambiguity in ground truth.
7 Difficulty Tiers
Each task is tested at multiple difficulty levels. The four primary tiers:
- Clean — pristine, well-formatted input. The baseline.
- Noisy — OCR artifacts, typos, field reordering.
- Adversarial — truncation, missing fields, conflicting data.
- Degenerate — all mutators chained. The stress test.
Three additional tiers combine mutators in different ratios. A model's accuracy at the Degenerate tier reveals how robust its understanding is versus brittle pattern matching.
Scoring
For extraction tasks (text→json, image→json): accuracy = percentage of fields correctly extracted, compared field-by-field against the synthetic ground truth.
For classification tasks: accuracy = exact match on the correct label.
For code generation tasks: accuracy = percentage of test cases that pass when the generated code is executed in a sandboxed environment.
For translation and summarization: accuracy is measured via anchor term preservation and structural fidelity scoring.
Update Frequency
The benchmark pipeline runs continuously. New data points are generated every 3 seconds. The public leaderboard reflects a rolling 14-day window of results, aggregated and refreshed every 4 hours.
Why Synthetic Data?
Human-labeled benchmarks have three problems: they're expensive to maintain, they go stale, and they leak into training data. Our synthetic generators produce unlimited, fresh, deterministic test cases that no model has seen before — because they didn't exist until the test ran.
This approach is called Crystallized Intelligence (CI). The test logic is frozen and deterministic; only the LLM's response varies. This eliminates evaluation ambiguity and ensures reproducible results.
Limitations
Synthetic benchmarks measure specific, well-defined tasks. They do not measure creativity, conversational ability, instruction following on open-ended prompts, or subjective quality. If your use case is "write me a marketing email," this benchmark won't help. If your use case is "extract invoice fields from a PDF," it will.