LPBench

Lean 4 · Combinatorics · Graph theory

LPBench

Research-level mathematics benchmark, verified by Lean.

LPBench, formerly ProofBench, evaluates whether AI systems can generate complete, mechanically checked proofs of meaningful mathematical statements. Its philosophy is simple: progress in AI mathematics should be measured not only by raw performance, but by the ability to reason through deep ideas and formalize expert mathematical insights. By expressing these insights as precise Lean 4 statements, LPBench enables objective comparison across AI systems and advances reliable proof generation and verification.

Technische Universität Bergakademie Freiberg · Universität Hamburg

P042.lean 1 / 7
declaration uses 'sorry'
The Riemann Hypothesis · open Open in problem list →
problems in Lean 4
118
open conjectures
32

Why LPBench

Built to measure real mathematical progress

Competition benchmarks are close to saturation, and informal proofs are expensive to grade reliably. LPBench takes a different route: research-level problems, exact formal statements and fully automatic verification.

Research-level by design

Problems are chosen by mathematicians active in the field: recent theorems, classical results and conjectures that are still open. No competition puzzles.

Verified, not graded

There is no human grader and no LLM judge. A submission counts only if it compiles against the untouched statement and passes an independent proof checker.

Statements you can trust

Every formal statement is checked against results that should imply it and consequences it should imply, so mis-formalizations are caught before any model sees them.

A level playing field

Every model runs in the same sealed sandbox with the same tools, budget and session limit, and every run is logged so it can be reproduced.

The problem set

118 problems, from classical theorems to open conjectures

Every problem is a single Lean file with one theorem whose proof is left as sorry. The collection is anchored in graph theory and reaches into matroids, extremal set theory, number theory and geometry.

  • Open32unsolved; prove it or its negation
  • Research65theorems from the research literature
  • Calibration21classical results that test the pipeline
Each square is one problem. Hover to identify it, click to open it.

Theorem sandwich

Statements you can trust

One misplaced quantifier can turn a deep conjecture into a trivial exercise, or into a false statement. To catch such slips before any model sees a problem, each statement is bracketed by two independent layers of Lean code.

Upper layer

Key lemmata, or a stronger known result

Some may stay unproved. Proving all of them would solve the problem.

⟹

The statement

theorem Pxxx : … := by sorry

⟹

Lower layer

Easy corollaries

Each one derived from the statement in Lean.

Evaluation

Every model, the same conditions

AI systems are evaluated in one shared harness built on Inspect AI, not inside each vendor’s own agent product, so results stay comparable across models and reproducible over time.

  1. 01

    A sealed workspace

    Each attempt runs in its own container with Lean, Mathlib and the statement library. No internet, no credentials, no route to reference solutions.

  2. 02

    Real research tools

    A shell, the Lean language server, Python, SageMath and a SAT solver. Web search only in dedicated runs, through one shared provider.

  3. 03

    Fresh sessions

    Up to ten sessions per problem. Each starts from a clean context and inherits only the current files and a structured handoff note.

  4. 04

    Two independent checks

    Lean must compile the untouched statement using standard axioms only. Comparator then rebuilds and replays the proof in isolation.

Anatomy of an attemptillustrative run, not a real result
    Logged for every attempt: verdictsessions usedtokens in & outcost total & working timetool callsproof length

    Open source

    Run LPBench yourself

    The problem statements, the evaluation harness and the HPC sandbox will be released together with the paper.

    Landscape

    Where LPBench fits

    Benchmarks for AI mathematics differ in what they ask for, a final answer or a full proof, and in how the answer is checked. At the research level, formal and proof-based evaluation is still rare.

    FormatCompetitionUndergraduateGraduateResearch

    More from LPBench

    Keep exploring

    The LPBench Workshop

    The first steps of the project were taken at this workshop. Approximately twenty participants took part, including research mathematicians with a combinatorics background and a smaller group of Lean experts. The objectives were to lay down the standards that a Lean-based benchmark of research-level mathematics should meet, to formalize a current research result in Lean 4 to serve as a test problem for existing AI systems, and to bring together experts in pure mathematics, interactive theorem proving and machine learning.

    22 September – 3 October 2025 · Technische Universität Bergakademie Freiberg, Germany

    Pencil sketch of two QED boxes with arms and legs sitting on a park bench; one is reading a book.
    weeks in Freiberg
    2
    mathematicians and Lean experts
    ≈ 20
    recent research result in focus
    1

    The mathematics

    The workshop centred on a question from infinite graph theory. Call a graph H ubiquitous if every graph that contains n disjoint copies of H as minors, for each finite n, must also contain infinitely many disjoint copies. In 2002 Thomas Andreae conjectured that every locally finite graph is ubiquitous. In 2023 Johannes Carmesin showed that this fails by constructing a counterexample (Journal of Combinatorial Theory, Series B 162).

    Carrying that construction all the way into Lean 4 was the shared goal. The argument fills only a few pages on paper, but it rests on infinite graphs, minors and disjoint unions, areas where Mathlib offers little ready-made support.

    theorem exists_countable_simpleGraph_not_isUbiquitous :
              ∃ (W : Type) (_ : Countable W) (H : SimpleGraph W), ¬ IsUbiquitous H := by
            refine ⟨_, inferInstance, barbellRay, fun h ↦ ?_⟩
            exact not_foldedUnion_barbellRay_nat_isMinor_decoratedGrid <|
              h _ _ foldedUnion_barbellRay_fin_isMinor_decoratedGrid

    An excerpt of the Lean 4 code written during the workshop.

    How we worked

    1. Mixed teams

      Participants formed teams that paired combinatorial expertise with Lean experience, and each team took on part of the argument.

    2. Smaller units

      As the work progressed, the teams shrank. Small groups moved more easily between the mathematics and its formal verification.

    3. Lean experts on call

      The Lean specialists gradually left their fixed teams and became a shared resource, joining whichever group was stuck.

    4. A tested workflow

      Along the way the group developed procedures for splitting a complex proof into manageable formalization tasks, and gave the theorem sandwich its first trial.

    What we learned

    Formalization is hard, even for experts

    Even with a strong team, turning a recent research proof into checked Lean code took sustained effort. The gap between a convincing argument and a verified one is real.

    Statements are where errors hide

    Writing Lean statements that say exactly what the mathematics says was the most delicate step. Small slips were easy to make and hard to spot, which is why LPBench now guards every statement with a theorem sandwich.

    Plan before you formalize

    Graph theory in Mathlib is still thin, so much time went into basic definitions. Mapping out the proof and its dependencies in advance with a blueprint catches problems early.

    Why it mattered

    The workshop showed that research-level formalization is a workable, and demanding, yardstick for AI systems. It validated the pipeline that LPBench uses to construct its problems and was the first real test of the theorem sandwich, bringing pure mathematicians, Lean users and machine-learning researchers to the same table.

    Participants

    • Justin Asher
    • Nathan Bowler Hamburg
    • Johannes Carmesin TU Freiberg
    • Joseph Devine TU Freiberg
    • Sarah Frenkel Birmingham
    • Matthias Hamann Hamburg
    • Julian Holstein Hamburg
    • Nicolai Kraus Nottingham
    • Matt Kroeker TU Freiberg
    • Jan Kurkofka TU Freiberg
    • Nicola Lorenz Hamburg
    • Aref Mohammadzadeh Nottingham
    • Felix Pernegger Bonn
    • Tim Planken TU Freiberg
    • Auguste Poiroux EPFL
    • Martin Schneider TU Freiberg
    • Jacob Stegemann Hamburg
    • Will Turner TU Freiberg
    • Martin Winter TU Berlin

    Team

    The people behind LPBench

    • Vaibhav BajpaiTechnische Universität Bergakademie Freiberg
    • Nathan BowlerUniversität Hamburg
    • Johannes CarmesinTechnische Universität Bergakademie Freiberg
    • Jinghua GroppeTechnische Universität Bergakademie Freiberg
    • Sven GroppeTechnische Universität Bergakademie Freiberg
    • Utku OkurTechnische Universität Bergakademie Freiberg

    LPBench is an academic initiative of the Institute of Discrete Mathematics and Algebra at Technische Universität Bergakademie Freiberg, together with the Department of Mathematics at Universität Hamburg.

    Contribute

    Propose a problem

    1. Send us a result, a conjecture or a question from your area of research.
    2. We formalize it in Lean 4, with you or independently, and build its theorem sandwich.
    3. The statement joins the benchmark.
    Propose a problem

    Contribute

    Propose a problem

    Send us a result, a conjecture or a question from your area of research. We will reply by email.

    About you
    Your problem

    Your details are used only to handle this proposal. The form is delivered by email through FormSubmit.