The rule did not reach the pipeline.

Picture a page from a pipeline specification, the kind an engineer writes before a training job runs. It lists what the evaluator, the component that scores each attempt, is allowed to read: the final answer, the tool calls, the tool outputs. One line has been struck through, and beside it someone has written in the margin: do not pass this to the evaluator. The struck line is the model's reasoning trace. Now picture the log from a run built to that specification. The reasoning trace field is absent, as promised. But a few lines down, inside a tool output the evaluator is allowed to read, sits a sentence from the reasoning trace, word for word. The rule was followed, and the content arrived anyway. The scene is invented. The failure is not.

On Thursday, May 7, OpenAI's alignment team published an account of how some of its own training runs had broken one of its own rules.¹ The company has a policy against grading a model's chain of thought, the written reasoning a model produces on the way to an answer, while training its reasoning models. Yet three components that computed rewards had been given that reasoning text, in a small share of the training samples for several released models. The same day, Buck Shlegeris of Redwood Research, who had reviewed a draft of OpenAI's account before publication, published a review that mostly agreed with its reassuring conclusions and set out what remained uncertain.²

My argument is that a data-use rule becomes dependable only when someone checks the paths the data actually takes. A declared data-use rule states an intention: this input must not reach that process. An observed data flow is evidence about what happened when the system ran. The two can disagree without anyone breaking the rule on purpose, because a forbidden input can reach a process by an indirect route that the obvious configuration never mentions. Governance needs both, and it needs the second to test the first. This is not a claim that OpenAI's models were harmed, or that its policy failed. The published evidence points away from the first, and the policy is what made the problem visible at all.

Why the reasoning had to stay out

The rule exists because a reasoning trace is evidence. When a model works through a problem in writing before it acts, a person or a second program can read that writing and catch a plan to cut a corner or deceive the user. Researchers call this property monitorability: the reasoning is useful to a monitor because nothing in training has rewarded it for looking a certain way.

Grading the reasoning changes that. If the reward depends on what the reasoning says, the model is paid for reasoning that pleases the grader, which need not be reasoning that is accurate. OpenAI's example is a reward for aligned-looking reasoning that teaches a model to leave out the incriminating thought when it misbehaves.¹ The authors write that with enough pressure of this kind they would expect monitorability to degrade. That is a research concern about a mechanism. Whether this particular, accidental, and fairly small amount of pressure did any measurable damage is a separate question, and it has a separate answer, which comes later.

The three components that saw the reasoning were each built for a sensible purpose: one rewarded useful work, one penalized unnecessary questions asking the user for confirmation, and one penalized the model when a prompt injection succeeded. Each one, OpenAI explains, could in principle have taught the model to shade its reasoning: to overstate its usefulness, to make a confirmation question sound necessary, or to hide reasoning about an injection.¹

How content slips past a named field

The general failure is older than machine learning. A rule that says "do not send field X to process Y" is enforced by removing field X. But a field is a container, and the rule cares about what is inside it. Once the content is copied into a log, summarized into a note, embedded in another message or cached for reuse, it travels under a different name, and a filter that looks for the old name lets it through.

OpenAI's account includes a case of exactly this shape, separate from the three components. During training, some models could read their own earlier reasoning through tools, so the reasoning appeared inside tool outputs, and those outputs were later graded. The company's detection system caught this before it affected any deployed model.¹ The authors draw the general lesson themselves: "CoT can leak into reward inputs through indirect paths."¹

Computer security named the remedy half a century ago. In their 1975 paper on protecting information, Jerome Saltzer and Michael Schroeder set out a principle they called complete mediation: "Every access to every object must be checked for authority." Beside it they put fail-safe defaults: base access decisions on permission rather than exclusion.³ The analogy has limits. Access control decides who may read an object; a training rule is about what may influence a score, which is a property of content and use rather than of identity. But the lesson carries over. A rule is enforced where the data is used, every time, not upstream where someone once configured it to be left out. And a list of what is allowed in is sturdier than a list of what is kept out, although even an allow list of fields still governs labels rather than contents.

How the problem was found

The most instructive part of OpenAI's account is the detection. The company runs an automated system that scans its reinforcement learning runs for reasoning text appearing in the inputs to rewards, using pattern matching, and alerts the owner of any run where it finds some. The system was improved shortly after the release of GPT-5.4, and the improved version found the accidental grading.¹

Notice what kind of check that is. It does not read the configuration and confirm that the reasoning field is switched off. It looks at what actually reached the reward, and it looks for the content rather than the name. That is a check on the observed data flow, and it found what the configuration alone did not reveal. The authors are also candid about its reach: some reward inputs are non-standard and harder to check automatically.¹ A check on observed flow is only as good as the places it looks.

A test any team can run

Most organizations have rules of the same form: customer data must not reach the analytics vendor, test results must not reach the model being tested, a contractor's notes must not reach the bid committee. Each can be turned from a statement into a property of the pipeline that can be checked. What follows is a suggested practice, not a procedure OpenAI describes.

Choose a sentinel: a made-up, harmless phrase that appears nowhere else, never real customer or sensitive data. Place it in the forbidden source. Run the pipeline the way it really runs, with the same tools, retries and caches. Then search every input to the forbidden destination for the sentinel, including the indirect carriers: tool outputs, logs, caches, intermediate files, summaries and retrieved documents. If the phrase appears, you have found a route, and you know which one. If it does not, you have evidence, bounded by where you searched and how. Run the test again whenever the pipeline changes, because a new tool or a new cache is a new route.

The test belongs at the destination for the reason Saltzer and Schroeder gave. Upstream, there are many ways for content to leave a source. At the destination, there is one door to watch.

What the record should say

When a test finds a route, or an incident reveals one, the record of it should let someone outside the team tell the intended configuration from the observed operation. That takes a short list of fields: the version of the policy, the version of the pipeline, the path the data took, the control that detected it, the runs affected, the corrective action, and the uncertainty that remains.

OpenAI's account supplies most of those: the three components and their purposes, the affected models and the fraction of samples involved, the detection method, the fix to the affected reward pathways, and a list of new safeguards, including detection that looks beyond grader configuration and recurring stress tests.¹ Redwood's review shows what such a record is for. Shlegeris could check the claims against the evidence offered, and he could name what was missing. The scale of the training, he writes, is the most important fact the post leaves out, withheld because it is commercially sensitive, and some of his confidence rests on statements OpenAI staff made without sharing their justification.² His review was a critical reading of supplied evidence, not an independent rerun of training. A record cannot make an outside reviewer certain. It can make the reviewer's uncertainty specific.

What the investigation showed

It helps to keep three questions apart: whether the forbidden content reached the destination, how it got there, and what it did. The first two have clear answers here. The content arrived, through reward components that had been given the reasoning text. The third is harder, and OpenAI treated it as an empirical question rather than settling it by assertion.

For one component, the team reran the affected portion of training with the reasoning removed and compared the two models; it found no evidence that grading the reasoning had changed the targeted behavior, and no significant drop on its monitorability tests. For another, it recomputed the rewards with and without the reasoning and found differences comparable to the ordinary disagreement between two graders that never saw it. For the third, it ran no targeted analysis.¹ The headline is careful: the analyses showed no clear reduction in monitorability, though the authors write that they cannot rule out effects that are harder to measure. In a separate set of stress tests, one setting did show a meaningful drop in a monitor's ability to catch the behavior it was looking for.¹

Shlegeris writes that he mostly agrees with the analysis, and he regards it as moderately strong evidence against an important loss of monitorability. His main reservation is that a monitorability score could miss a narrow kind of damage, such as a rare class of thoughts that the model learned to leave unsaid.² That is the right shape for a conclusion. The exception was real, its mechanism is understood, and its demonstrated impact is small within the limits of what was measured.

The same discipline applies to any organization that discovers a rule did not reach its pipeline. An exception should start two investigations: what it did, and which similar paths exist. It should not be read as proof that the whole system is unsafe, and it should not be closed as harmless because nothing broke.

Objections, considered

Observed flow is only as complete as the instruments. True. A trace that does not cover a route cannot find content on it, and OpenAI says as much about its non-standard reward inputs. That is why the record should state where the search looked, so that a clean result is read as clean within those places and no further.

Exact matching misses transformations. A sentence that has been paraphrased, translated or summarized will not match its original, and broader matching on meaning raises false alarms. Both kinds of error are real. A sentinel can be chosen to survive the common transformations, and a team can report both its matching method and its false-alarm rate rather than pretend either is zero.

A rule reduces risk even before it is perfectly enforced. It does. Without the rule there would have been no scan, no alert, and no investigation. The argument here is not that declared rules are worthless. It is that a rule without a test describes the system its authors meant to build.

An exception may not matter much. Its significance depends on how much forbidden content arrived and how strongly the process was pushed to use it. A handful of samples in an enormous run may be negligible, or may not; Shlegeris's point about undisclosed training scale is exactly this. The answer is to report occurrence, mechanism and demonstrated impact separately, so that a small exception is neither inflated nor waved away.

The rule describes the intention; the trace shows the route.

Before you approve the next data-use rule

• Where is it checked? At the destination, by content, or only at the source, by the field's name?
• Which indirect paths have been tested? Name them: tool outputs, logs, caches, intermediate files, summaries.
• What would the test miss? State the matching method, where it searched, and what it cannot see.
• When did it last run? Tie the test to every change in the pipeline, not to the date the rule was written.
• What happened to the last exception? Ask for the versions, the route, the affected runs, the fix and the uncertainty that remains.

Back to the specification with the struck-through line. Keep the line, and keep the note in the margin: they say what the organization intends. But attach a test to them: a sentinel planted at the source, a search at the evaluator, the date the search last ran and what it found. Then the margin note is no longer only a promise about the pipeline. It is a property of it.