LPBench

Benchmark Evaluation

QEDRank is a benchmark evaluation platform for formal mathematics. It provides two complementary workflows: evaluate how effectively language models solve a selected corpus of formal mathematics problems, with Lean-verified, statistically analysed results, or compare different benchmark datasets to understand their scope, difficulty, coverage, and model performance.

Lean 4 · Mathlib · every answer is checked by the pinned Lean 4 toolchain; no composite score is ever computed

models in the roster: Claude, GPT, Qwen, GLM, DeepSeek
10
benchmarks to compare with, from miniF2F to LeanEval
9
benchmark targets: main theorem · upper · lower · disproof
4

Two workflows

Workflow 1

Evaluate Models

Evaluate one or more language models on a selected benchmark corpus. Generate Lean 4 formalizations, verify them with the Lean compiler, and compare model performance across correctness, auditability, reliability, efficiency, and difficulty.

  • Verified proof rate (pass@1) — primary metric
  • Audit-clean rate — proofs passing axiom and forbidden-import checks
  • Lean compilation success rate
  • Statement fidelity rate
  • Statement-match classification
  • Pass@k performance for k = 1–16
  • Attempt statuses and failure classes
  • Abstention, timeout, unresolved, and infrastructure-error rates
  • Latency, Lean compilation time, and token usage
  • Estimated cost per attempt and per verified proof
  • Performance by difficulty and subject area
  • Performance on the common-problem subset
  • Performance of different LLMs on your own dataset (e.g. LPBench)
  • 95% confidence intervals and exact denominators

Models: Claude Fable 5.1, Claude Opus 5.5, GPT-6 Astra, GPT-6 Sol, GPT-5.6 Terra, GPT-5.4, and Qwen3, Qwen3.5, GLM-5.3 Flash and DeepSeek V4 Flash served by GWDG Chat AI.

Open Evaluate Models ↗

Workflow 2

Compare Benchmarks

Compare your benchmark corpus with other mathematical formalization benchmarks to understand how they differ in scale, scope, difficulty, coverage, reliability, and model performance. The analysis also examines dataset overlap and duplication, theorem categories, formalization environments, confidence intervals, and the potential for future progress across each benchmark.

  • Number of problems as published
  • Number of problems measured
  • Release date, source, task type, and judging method
  • Mathematical domain and MSC subject coverage
  • Largest subject area
  • Difficulty-level distribution
  • Research-level and open-problem share
  • Theorem, definition, lemma, and construction-problem types
  • Formalization language, Lean and Mathlib versions
  • Theorem signature complexity
  • Statement length, including used definitions
  • Longest 10% of statements by token length
  • Theorems per file and definitions per file
  • Reference-proof availability and quality
  • Proof length, proof-term size, tactics, and automation share
  • Statements with contradictory or vacuous hypotheses
  • Linter warnings and reference defects
  • Dataset overlap and duplication
  • Model verified-proof rate by benchmark
  • Coverage differences between models and benchmarks
  • Headroom for future progress

Benchmarks: miniF2F, PutnamBench, FormalMATH, MSC-180, ProofNet, CombiBench, LeanEval, AMBER and FormalProofBench, or any other benchmark by its URL.

Open Compare Benchmarks ↗

More tools

How results are reported

-- Prove that the square root of 2 is irrational.
theorem sqrt_two_irrational : Irrational (√2) := by
  sorry

An illustrative challenge file. The statement is fixed; a model has to replace the sorry with a proof that Lean accepts.

Try QEDRank

Evaluate models or compare benchmarks in the browser at qedrank.vercel.app.

Open QEDRank ↗