Research-level by design
Problems are chosen by mathematicians active in the field: recent theorems, classical results and conjectures that are still open. No competition puzzles.
Lean 4 · Combinatorics · Graph theory
Research-level mathematics benchmark, verified by Lean.
LPBench, formerly ProofBench, evaluates whether AI systems can generate complete, mechanically checked proofs of meaningful mathematical statements. Its philosophy is simple: progress in AI mathematics should be measured not only by raw performance, but by the ability to reason through deep ideas and formalize expert mathematical insights. By expressing these insights as precise Lean 4 statements, LPBench enables objective comparison across AI systems and advances reliable proof generation and verification.
Technische Universität Bergakademie Freiberg · Universität Hamburg
Why LPBench
Competition benchmarks are close to saturation, and informal proofs are expensive to grade reliably. LPBench takes a different route: research-level problems, exact formal statements and fully automatic verification.
Problems are chosen by mathematicians active in the field: recent theorems, classical results and conjectures that are still open. No competition puzzles.
There is no human grader and no LLM judge. A submission counts only if it compiles against the untouched statement and passes an independent proof checker.
Every formal statement is checked against results that should imply it and consequences it should imply, so mis-formalizations are caught before any model sees them.
Every model runs in the same sealed sandbox with the same tools, budget and session limit, and every run is logged so it can be reproduced.
The problem set
Every problem is a single Lean file with one theorem whose proof is left as sorry. The
collection is anchored in graph theory and reaches into matroids, extremal set theory, number theory
and geometry.
By primary MSC class: 84 graph theory · 8 designs & matroids · 6 extremal combinatorics · 20 other
Browse all 118 problems →Theorem sandwich
One misplaced quantifier can turn a deep conjecture into a trivial exercise, or into a false statement. To catch such slips before any model sees a problem, each statement is bracketed by two independent layers of Lean code.
Upper layer
Key lemmata, or a stronger known result
Some may stay unproved. Proving all of them would solve the problem.
The statement
theorem Pxxx : … := by sorry
Lower layer
Easy corollaries
Each one derived from the statement in Lean.
Evaluation
AI systems are evaluated in one shared harness built on Inspect AI, not inside each vendor’s own agent product, so results stay comparable across models and reproducible over time.
01
Each attempt runs in its own container with Lean, Mathlib and the statement library. No internet, no credentials, no route to reference solutions.
02
A shell, the Lean language server, Python, SageMath and a SAT solver. Web search only in dedicated runs, through one shared provider.
03
Up to ten sessions per problem. Each starts from a clean context and inherits only the current files and a structured handoff note.
04
Lean must compile the untouched statement using standard axioms only. Comparator then rebuilds and replays the proof in isolation.
Open source
The problem statements, the evaluation harness and the HPC sandbox will be released together with the paper.
Results
The first evaluation results will be published here together with the paper. An open problem counts as solved when a system proves either the statement or its negation.
| Rank | Model | Web search | Open /33 | Research /63 | Calibration /21 | Total | Cost |
|---|
Landscape
Benchmarks for AI mathematics differ in what they ask for, a final answer or a full proof, and in how the answer is checked. At the research level, formal and proof-based evaluation is still rare.
| Format | Competition | Undergraduate | Graduate | Research |
|---|
More from LPBench
Sister project
The Ten Challenges comprise the first ten problems of LPBench, which are parameterized open problems in combinatorics.
Visit Ten Challenges →
Freiberg, autumn 2025
Two weeks, about twenty mathematicians and Lean experts, and a recent research result taken into Lean together. The first real test of the LPBench method.
Read about the workshop ↓The first steps of the project were taken at this workshop. Approximately twenty participants took part, including research mathematicians with a combinatorics background and a smaller group of Lean experts. The objectives were to lay down the standards that a Lean-based benchmark of research-level mathematics should meet, to formalize a current research result in Lean 4 to serve as a test problem for existing AI systems, and to bring together experts in pure mathematics, interactive theorem proving and machine learning.
The workshop centred on a question from infinite graph theory. Call a graph H ubiquitous if every graph that contains n disjoint copies of H as minors, for each finite n, must also contain infinitely many disjoint copies. In 2002 Thomas Andreae conjectured that every locally finite graph is ubiquitous. In 2023 Johannes Carmesin showed that this fails by constructing a counterexample (Journal of Combinatorial Theory, Series B 162).
Carrying that construction all the way into Lean 4 was the shared goal. The argument fills only a few pages on paper, but it rests on infinite graphs, minors and disjoint unions, areas where Mathlib offers little ready-made support.
theorem exists_countable_simpleGraph_not_isUbiquitous :
∃ (W : Type) (_ : Countable W) (H : SimpleGraph W), ¬ IsUbiquitous H := by
refine ⟨_, inferInstance, barbellRay, fun h ↦ ?_⟩
exact not_foldedUnion_barbellRay_nat_isMinor_decoratedGrid <|
h _ _ foldedUnion_barbellRay_fin_isMinor_decoratedGrid
An excerpt of the Lean 4 code written during the workshop.
Participants formed teams that paired combinatorial expertise with Lean experience, and each team took on part of the argument.
As the work progressed, the teams shrank. Small groups moved more easily between the mathematics and its formal verification.
The Lean specialists gradually left their fixed teams and became a shared resource, joining whichever group was stuck.
Along the way the group developed procedures for splitting a complex proof into manageable formalization tasks, and gave the theorem sandwich its first trial.
Even with a strong team, turning a recent research proof into checked Lean code took sustained effort. The gap between a convincing argument and a verified one is real.
Writing Lean statements that say exactly what the mathematics says was the most delicate step. Small slips were easy to make and hard to spot, which is why LPBench now guards every statement with a theorem sandwich.
Graph theory in Mathlib is still thin, so much time went into basic definitions. Mapping out the proof and its dependencies in advance with a blueprint catches problems early.
The workshop showed that research-level formalization is a workable, and demanding, yardstick for AI systems. It validated the pipeline that LPBench uses to construct its problems and was the first real test of the theorem sandwich, bringing pure mathematicians, Lean users and machine-learning researchers to the same table.
Team
LPBench is an academic initiative of the Institute of Discrete Mathematics and Algebra at Technische Universität Bergakademie Freiberg, together with the Department of Mathematics at Universität Hamburg.
Contribute