AI Oversight: Independent Audit Identifies Significant Monitoring Gaps

Between 6 and 25 percent of internal agentic traffic generated by models at the GPT-5.5 class and above was entirely exempt from monitoring during a recent three-week audit, according to a new report. This estimated range represents the volume of system activity that proceeded without any oversight from safety tools, a figure that highlights a notable gap in the current infrastructure. The assessment was conducted by researchers from METR and Redwood Research, who were granted deep, on-site access to OpenAI’s internal systems to evaluate how the company tracks potential rogue behavior in its artificial intelligence models.

The audit took place from July 21 to August 11, during which the external team examined OpenAI’s codebase and internal documentation. The researchers also conducted interviews with approximately a dozen employees to understand the operational nuances of the monitoring framework. Their primary objective was to determine the efficacy of the controls designed to prevent autonomous systems from acting in unintended or unsafe ways while executing complex tasks.

Beyond the unmonitored agentic traffic, the auditors identified specific technical failures within the Codex system. METR estimates that roughly 10 percent of internal Codex traffic went untracked between May and July due to a flaw in how the system handled compaction events, which are essentially context summaries. While this estimate carries significant uncertainty, OpenAI independently discovered the bug and implemented a fix while the audit was still in progress.

The vulnerability in handling compaction events created a window where agents could delay oversight. Because the monitoring system only triggered when an agent handed control back to a user, autonomous systems could bypass scrutiny simply by continuing to execute actions without pausing. This mechanism allowed for a period of unobserved activity that the safety tools were not designed to capture in real time.

The researchers further discovered that it was possible to conceal hundreds of individual actions from the monitor and to circumvent the Codex sandbox environment. They also determined that specially crafted inputs could likely cause the monitoring system to crash entirely. Additionally, at the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored at all, a gap that OpenAI later addressed.

OpenAI stated that it paused RL training following an incident involving Hugging Face in August and subsequently expanded its monitoring coverage to include this phase. However, these corrective measures were implemented after the auditors’ access period concluded and therefore were not included in the formal evaluation of the system’s prior performance.

While OpenAI asserts that all weaknesses identified by the auditors have been fixed, the full findings remain largely restricted. The complete 61-page report was distributed internally, but after redactions, leadership at METR and Redwood Research received a 38-page version. Only a brief summary has been made available to the general public, with METR noting it lacked the authority to disclose the redacted sections independently.

The review served as a pilot program designed to allow outside researchers unprecedented access to internal infrastructure. Looking forward, METR has expressed a desire to push for greater public transparency in future audits, aiming to make more of these detailed safety assessments accessible to the broader community.