Benchmark Evaluation
QEDRank is a benchmark evaluation platform for formal mathematics. It provides two complementary
workflows: evaluate how effectively language models solve a selected corpus of formal mathematics
problems, with Lean-verified, statistically analysed results, or compare different benchmark datasets to
understand their scope, difficulty, coverage, and model performance.
Lean 4 · Mathlib · every answer is checked by the pinned Lean 4 toolchain; no composite score is ever computed
- models in the roster: Claude, GPT, Qwen, GLM, DeepSeek
- 10
- benchmarks to compare with, from miniF2F to LeanEval
- 9
- benchmark targets: main theorem · upper · lower · disproof
- 4
Two workflows
∀
Workflow 1
Evaluate Models
Evaluate one or more language models on a selected benchmark corpus. Generate Lean 4
formalizations, verify them with the Lean compiler, and compare model performance across correctness,
auditability, reliability, efficiency, and difficulty.
- Verified proof rate (pass@1) — primary metric
- Audit-clean rate — proofs passing axiom and forbidden-import checks
- Lean compilation success rate
- Statement fidelity rate
- Statement-match classification
- Pass@k performance for k = 1–16
- Attempt statuses and failure classes
- Abstention, timeout, unresolved, and infrastructure-error rates
- Latency, Lean compilation time, and token usage
- Estimated cost per attempt and per verified proof
- Performance by difficulty and subject area
- Performance on the common-problem subset
- Performance of different LLMs on your own dataset (e.g. LPBench)
- 95% confidence intervals and exact denominators
Models: Claude Fable 5.1, Claude Opus 5.5, GPT-6 Astra, GPT-6 Sol, GPT-5.6 Terra, GPT-5.4, and Qwen3, Qwen3.5, GLM-5.3 Flash and DeepSeek V4 Flash served by GWDG Chat AI.
Open Evaluate Models ↗
∃
Workflow 2
Compare Benchmarks
Compare your benchmark corpus with other mathematical formalization benchmarks to understand how they
differ in scale, scope, difficulty, coverage, reliability, and model performance. The analysis also
examines dataset overlap and duplication, theorem categories, formalization environments, confidence
intervals, and the potential for future progress across each benchmark.
- Number of problems as published
- Number of problems measured
- Release date, source, task type, and judging method
- Mathematical domain and MSC subject coverage
- Largest subject area
- Difficulty-level distribution
- Research-level and open-problem share
- Theorem, definition, lemma, and construction-problem types
- Formalization language, Lean and Mathlib versions
- Theorem signature complexity
- Statement length, including used definitions
- Longest 10% of statements by token length
- Theorems per file and definitions per file
- Reference-proof availability and quality
- Proof length, proof-term size, tactics, and automation share
- Statements with contradictory or vacuous hypotheses
- Linter warnings and reference defects
- Dataset overlap and duplication
- Model verified-proof rate by benchmark
- Coverage differences between models and benchmarks
- Headroom for future progress
Benchmarks: miniF2F, PutnamBench, FormalMATH, MSC-180, ProofNet, CombiBench, LeanEval, AMBER and FormalProofBench, or any other benchmark by its URL.
Open Compare Benchmarks ↗
More tools
-
Results
Evaluation
Explore a finished run: verified proof rates, statement fidelity, Pass@1 and Pass@k performance, error patterns, latency, token usage, cost, and results by difficulty and subject area. A run is loaded from the local run server or from its result files.
Open Evaluation ↗
-
Landscape
Benchmark Landscape
LPBench and the Lean benchmarks on one chart: benchmark size against the best reported success rate. Statement statistics are measured with one protocol for every benchmark; sizes and best results come from the original papers or public leaderboards, with links to the sources.
Open Benchmark Landscape ↗
-
Single problem
Convert a Problem
Send one statement to every selected model and compare the Lean 4 answers side by side, with latency, tokens and your own rating. Answers are not compiled here; to verify them with Lean, use Evaluate Models.
Open Convert a Problem ↗
How results are reported
- A challenge file states the theorem and leaves its proof as
sorry. That sorry is the task, and every answer is checked against the untouched statement.
- Verified success is the strict primary endpoint: the answer compiles, uses no
sorry and no new axioms, and leaves the statement unchanged.
- Every rate is reported as numerator over eligible denominator at pass@1–16, with a 95% paper-cluster bootstrap interval.
- Statement fidelity, compilation and audit are reported alongside verified success and are never folded into one score. No weighted composite is computed.
-- Prove that the square root of 2 is irrational.
theorem sqrt_two_irrational : Irrational (√2) := by
sorry
An illustrative challenge file. The statement is fixed; a model has to replace the sorry with a proof that Lean accepts.