Human in the Loop AI: What OpenAI's Math Release Teaches About Checking Machine Work

Human in the loop AI sounds safe until you ask what the human checks. On October 6, OpenAI put hundreds of mathematics manuscripts on GitHub, produced by an internal model.

On October 7, its history file recorded that “a sign error invalidates a stabilization-trace cancellation argument and the construction used by two dependent papers.” Three manuscripts were withdrawn and fourteen more were revised.

The history file shows what a visible correction looks like. The record names the error, the papers it reached and what was done about them.

I cannot say whether the remaining results hold. I can say this record is the part of the process a marketer can copy.

Now ask what your own version looks like. Your AI wrote a campaign report this week. Who checked it, with what, and where is the record?

I am not a mathematician and I cannot judge the proofs. What I can read is the process around them.

The facts below come from OpenAI’s announcement, the repository’s README, history file, formalization catalogue and overview, and from one arXiv audit. I checked the quotations and figures against saved copies of those pages on October 14, 2026.

Key Takeaways

• OpenAI withdrew three manuscripts on October 7, the day after publishing, after a sign error broke an argument two other papers used. It also revised 14 others and wrote down the reasons.
• Two formalization figures do not obviously match. The README says about 42% of top-line results are formalized and the history file gives 300 of 719, while the catalogue file lists 173 papers. The repository does not say how they relate.
• A machine check verifies the statement as written in the machine. Whether that statement says what the paper claims is still a human question.
• Checkers can be wrong too. An arXiv audit of an earlier OpenAI set records an apparent error that was withdrawn once a lost overbar was recovered from the source.
• Human in the loop AI works with five parts: inputs the human can see, a rule to reject, a consequence of rejecting, a machine check first, and a record.
• When output is cheap, checking is the bottleneck. Budget for it first.

What OpenAI Released

OpenAI’s announcement is dated October 6, 2026. The repository holds “mathematical manuscripts and supporting proof artifacts produced by an internal OpenAI model.” The README counts 719 manuscripts in 372 families as of October 14, and the Apache-2.0 license applies.

What OpenAI states about the work:

• The model was unreleased as of October 14. “The vast majority of results were obtained with the same procedure using an unreleased internal OpenAI model.”
• It was posed about 4,000 problems. The catalogue is what remained after “requiring an appropriate level of significance.”
• Each result used about three hours of ChatGPT Pro compute, on average. That is per result, not per problem.
• OpenAI consulted the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study on how to share the work. The post does not say the group checked the results.

The headline claims are big. Family 003 says every Dirichlet L-function, the Riemann zeta function included, has no zeros with real part above 7/8, which the overview PDF calls “resolving the quasi-Riemann hypothesis.” That is not the Riemann hypothesis.

Family 004 says no algorithm decides whether an integer-coefficient polynomial in an arbitrary number of variables has a rational zero, which it presents as resolving Hilbert’s tenth problem over the rationals. Family 107 claims a matrix multiplication exponent of at most 9/4 over the rationals.

I am reporting those as claims. I am not the person to confirm or doubt them.

The three withdrawn papers include one titled “The rational Hodge conjecture for products of K3 surfaces,” so the correction touched a major named conjecture. The README is unusually specific about what is unchecked.

“This collection includes results at different stages of verification. Not all have accompanying Lean formalizations.”

Then: “Some of the unformalized results could have issues. We will endeavor to fix any such issues quickly.”

Two Counts That Do Not Match

The README gives one count: “the repository has ~42% top-line results formalized.” The October 7 history entry says the update “brings the total percentage of top-line results formalized to 300 / 719 = ~42%.”

The README says 719 is the number of manuscripts. The formalization catalogue lists 173 papers with a formalized main result, by my count on October 14. I did not find a page that reconciles 300 with 173.

That is not an error. It is the reason to ask what a percentage counts.

A report that is “90% verified” may mean 90% of claims, 90% of pages or 90% of the claims somebody thought to check. Ask which one before you trust it.

What a Lean Check Does, and What It Does Not

Lean, in OpenAI’s words, is “a programming language that allows mathematical proofs to be checked by a computer.” When a proof is formalized in Lean and Lean accepts it, the machine has verified that the formal statement follows from its axioms.

In plain terms, the machine verifies the statement as written in the machine.

Whether that statement says what the paper claims it says is still a question for a human who reads both. I am explaining that principle here. OpenAI’s pages do not say it in those words.

A green tick from a machine means the machine agreed with itself. You still have to read what it agreed to.

A two-author audit preprint on arXiv assessed 18 chapter-specific reviews of the ten results OpenAI announced on 1 August 2026, an earlier set than the October collection. It reports that “no confirmed substantive mathematical error in a principal result remains in the examined assessments, although review depth varies and some dependencies remain partly checked.”

It flags Chapter 8 for a specialist’s request for “major revision of compressed analytic arguments,” and it argues that “confidence should combine formal checking, human reconstruction, independent mathematical use, and a public record supporting correction of both proofs and reviews.” I cite it for the method, not as a verdict on the mathematics.

Marketing has no Lean. Most of its claims can only be checked against a source, a date or a person, which is why the record matters more than the tool.

The Checker Can Be Wrong Too

The audit has a detail I keep coming back to. In Chapter 6, an apparent polarity error was “withdrawn after an overbar lost during PDF extraction was recovered from the typeset source.”

The abstract does not say who flagged it. On that reading, the problem came from the text extraction, not from the proof. A loss like this is how a checking step can invent an error.

My own pipeline has a version of this check. Reading agents extract quotations from saved source pages, and a script confirms that each quotation appears word for word in the page. For the research behind five recent articles, it accepted 295 quotations and rejected two that did not appear word for word in their pages.

Then a fact-checker reads the draft, and I re-check what it flags with a text search before I accept it, because long source lines caused false alarms. A skeptical reviewer runs at high effort, and translations get a native-speaker second pass that has found real errors. A checker is a tool with an error rate, and the error rate needs a check.

The reviewer should not be the author. In my CLAUDE.md article I wrote that you should put verification in “checks a machine can fail,” and use a separate reviewer where taste or stakes are high, “because a model tends to prefer its own results.”

Why Human in the Loop AI Is Often Theater

The phrase promises safety, so people stop asking what it means. A common answer is “a human looks at it.” That is a slogan.

Here is the definition I use. A real loop has five parts, and if any is missing you have a ritual.

1. The human sees the inputs. The sources, the data and the instructions as well as the output. You cannot judge a summary you cannot trace.
2. There is a rule for rejecting. What would make a reviewer say no? If nobody can answer, every output passes.
3. Rejecting has a consequence. The output is redone, the process is fixed, or the use case is stopped. A reject button that changes nothing is decoration.
4. A machine check runs first. Code checks links, numbers, quotations, formats and totals, so the human spends attention on judgment.
5. There is a record. A log of what was checked, by whom, and what was fixed. OpenAI’s history file is a public example.

A loop with only one part, a person who looks, becomes a rubber stamp once you add a deadline and a queue of 200 items.

A Verification Table for Marketing Output

“The AI got it wrong” hides three different failures. Wrong means the statement is false: an invented feature, a made-up statistic, a quotation nobody said. Stale means it was true when the model learned it or when the source was written: a price, a legal deadline, a product version.

Mismatched means the statement is true and answers a different question than the one you asked. The summary is accurate for last quarter, or the average is accurate for the wrong region. Only a human who knows the question catches this, so a reviewer needs the brief and not only the output.

You do not need Lean. You need the cheapest check that fails loudly for each type of output.

Some checks have no machine option. Those are the ones where a named human must own the answer.

A Worked Example: The Competitor Price Summary

This example is made up, so do not look for these competitors. A marketing team asks an AI to summarize the pricing pages of four competitors into one table for the sales deck.

What the AI returns: a clean table with plan names, monthly prices and three headline features per plan.

Machine check: a script fetches each pricing page, saves a copy and confirms that every price in the table appears on the saved page next to the plan name. Two prices fail. One competitor shows annual prices by default, and the AI reported them as monthly.

Human check: a person who knows the sales conversation looks at the table and notices the features compare a premium plan from one vendor with an entry plan from another. That is a mismatch, and no script would find it.

Record: the saved pages, the date and the reviewer’s name go in the folder next to the deck. When a competitor changes its pricing, the team knows which slide is stale.

A script and one reviewer can plausibly do this in an hour for four pages. Your first run will take longer.

How to Write a Reject Rule

A reject rule is one sentence a reviewer can apply without a debate. These work for marketing:

• Reject any figure that does not appear in the source export.
• Reject any quotation that does not appear word for word in the saved page.
• Reject any claim about law, price or availability without a primary source and a date.
• Reject any customer-facing sentence that promises a result, a deadline or a discount.
• Reject any translation whose numbers differ from the original.

Write the list once, put it where the reviewer works and update it when a mistake slips through. Each slipped mistake becomes a new rule. This is a process suggestion, not legal advice.

What I Would Copy From OpenAI’s Repository

Strip away the mathematics and the repository has three habits worth copying.

A history file with reasons. Each withdrawal names the papers and the cause, and the revisions are grouped by what was repaired. For AI-written content, keep a changelog that says what was fixed and why, not only that something changed.

Old versions stay reachable. OpenAI says “Corrections and revisions will be recorded as new versions, with previously released versions remaining accessible.” On this blog every article carries a date, and when I change a published piece I move the updated date.

Disclaimers where the doubt lives. OpenAI says which results are formalized and which could have issues. Do the same in your reports: mark the numbers that were recomputed and the ones taken from the AI at face value.

When Output Is Cheap, Checking Is the Bottleneck

OpenAI says an average result used about three hours of ChatGPT Pro thinking. It does not say what checking one takes, and I would not guess.

The marketing version is easier to see. An AI can write 50 product descriptions in a minute, and confirming that none invents a feature is not a minute of work.

Using AI for marketing multiplies what your best people do, and it also moves the bottleneck from making to checking. A team that does not budget for checking has not saved time. It has moved the risk to the customer.

A 30-Minute Verification Audit

Run this on one AI workflow this week.

1. Pick one recurring output that reaches customers or decisions: a report, a reply template, a product feed.
2. Write down who checks it today and what exactly they look at. If the answer is “they read it,” go on.
3. List the five parts of a real loop and mark which exist.
4. Add the cheapest machine check from the table above.
5. Write the reject rule in one sentence.
6. Start a log with one line per run: date, checker, what was fixed.
7. Review the log in a month and move the first manual check into code.

Frequently Asked Questions

What does human in the loop mean in AI?

It means a person takes part in the AI process, usually by reviewing or approving output. It only works when the person sees the inputs, has a rule for rejecting and a way to act on a rejection, and when a record exists.

Is human in the loop enough to make AI output safe?

Not by itself. Add machine checks for what code can verify, a separate reviewer for judgment, and a log that lets you correct the record later.

Can AI check its own work?

It can catch some errors, and it tends to favor its own results. Use a different reviewer, a rubric and checks a machine can fail.

Are OpenAI’s math results proven?

I cannot tell you, and this article is not about that. OpenAI says the collection has “results at different stages of verification,” that about 42% of top-line results are formalized in Lean by its count, and that some unformalized results “could have issues.” Whether the mathematics is right is a question for mathematicians.

How do I verify AI output at work?

Start with one workflow. Add a machine check, a rule for rejecting and a log, and put a named person on the checks that code cannot do.

My Verdict

A human in the loop AI process is only as good as its ability to say no. A loop that cannot reject is a ritual.

The card:

1. Show the human the inputs.
2. Write the reject rule.
3. Make rejecting change something.
4. Run machine checks first.
5. Keep a record, and let the record show corrections.

This page reflects OpenAI’s repository as of October 14, 2026. The collection is being updated, so check the history file before you rely on a count.

Want a second pair of eyes on how your team checks AI output, with a checklist matched to your workflows? Get in touch.