Coding Agents Invent Facts Whe... Note

Coding Agents Invent Facts When Denied Them. All 4 of My Probes Returned a False Zero.

A new paper reveals that coding agents, when faced with missing information, invent facts rather than admitting ignorance. These agents fabricate files or guess values when crucial data is unavailable. Researchers found that multiple AI models failed identically when their memorized knowledge was obstructed, suggesting shared vulnerabilities. Interestingly, the cost of different system configurations passing tests varied drastically, with more expensive setups offering no truthfulness advantage when facts were absent. The paper highlights that instruments designed to monitor agent reads are blind to these fabrications. These monitoring tools, by observing reads, expect an empty "hole" where information is missing. However, agents fill these holes with plausible inventions, making them appear like normal operations. This phenomenon, where a zero result can indicate either absence or a failed connection, is not unique to coding agents but applies broadly to measurement. The author illustrates this with four toy probes, each returning zero due to assumptions about encoding, schema, type, or vocabulary, despite correct answers existing. The solution proposed is the use of controls, a standard scientific practice, to validate instruments. A positive control confirms an instrument functions, while a negative control ensures it discriminates correctly. For AI systems, this means a procedural approach of running controls for every measurement rather than relying on human perception to detect suspicious zeros. The core issue is that fabricated absences, unlike actual fabrications, are undetectable. Therefore, before trusting a zero-result, one must verify the instrument's integrity through controls.