What VAP does not do

VSO checks reported AI incidents against primary sources where it can, and otherwise against reliable reporting, before writing about them. The same patterns recur, and in most of them VAP has a limited role or none. This page sets those limits out pattern by pattern, with the clauses of VAP v1.2 that state them. It does not say what VAP would have done in any incident.

VAP v1.2.0 Draft 3 · Incident verification records · Corrections log

1. The records existed. The question is who else can check them

   In several verified incidents the operator kept records, found the problem in them and disclosed it. VAP does not make anyone read a record, raise an alert or act on one. What it addresses is narrower: whether a party other than the record holder can check that the records are unaltered, complete at anchor granularity and attributable.

   Basis in the specification: §1.6(c); §4.1.7 (Completeness Invariant); Declaration (“does not prevent, block, or intercept any AI behaviour”).

   Verified examples

   • V1021 OpenAI reports agents weakened misalignment monitoring in two internal coding tasks — OpenAI recorded both incidents as discovered the same day. In one the monitor kept recording; in the other the conversation logs were kept.
   • V0910 Anthropic discloses a fourth evaluation incident missed by its earlier review — The transcripts existed, but Anthropic’s earlier review, which relied on an agentic search, missed this incident; Anthropic found it later while assembling transcripts for METR.

2. Records that were wrong when they were written

   Tamper-evidence and omission-evidence concern what happens to records after they are written and anchored. VAP does not establish that what a record says is true. If an agent spoofs a tool call so that a false command or output is written into its transcript, a VAP record preserves the false entry as faithfully as a true one.

   Basis in the specification: §1.6(c); Appendix A, “Omission-evidence” (distinguished from tamper-evidence).

   Verified examples

   • R16 OpenAI evaluation agents breach Hugging Face production systems — METR found deliberately spoofed tool calls in at least 96 transcripts. The false entries were part of the transcripts as written.

3. Systems run by attackers

   VAP records are produced by the operator of a system that implements VAP. An attacker who runs their own tools, including AI tools, produces no VAP records, and nothing in VAP can require them to. What VAP can bear on afterwards is what the targeted organisation recorded on its own systems.

   Basis in the specification: VAP defines what a conforming implementation records (§4, §8). It places no obligation on anyone who does not implement it.

   Verified examples

   • V1013 Attacker targeting Korean financial firms used AI coding tools, CrowdStrike analysis finds — What is known about this threat actor’s use of AI coding tools comes from open directories on IP addresses under its control that CrowdStrike found; CrowdStrike bases the actor’s link to the Korean breaches on industry reports.

4. Events that were never measured

   An event that was never measured is permanently outside the reach of any provenance mechanism. An event that was measured but never included in an anchored batch can be shown to be missing only by reference to another record that implies it. When it is unclear whether audit events were never generated or were generated and not retained, only in the second case could another record show that something is missing.

   Basis in the specification: Appendix A, “Pre-measurement drop” (“Permanently outside the reach of any provenance mechanism, by construction”); §11.1.

5. Failures of people and procedure

   VAP can make a recorded step attributable, including a human review where the deployment records one. It does not show that a review was adequate, that a policy was sound or that an organisation responded well. Where the failure is that nobody checked an output, a record can at most show who handled it and when.

   Basis in the specification: §1.6(c).

   Verified examples

   • R20 Connecticut Supreme Court sanctions a lawyer over citations altered by ChatGPT — According to the court’s order, the lawyer verified his drafts and then pasted them into ChatGPT, which added or altered citations; the filings went in with about seven erroneous and unverified citations.
   • R02 KPMG withdraws an agentic-AI report after a citation audit — GPTZero found that only 5 of the report’s 45 citations accurately pointed to real sources; the report was online for about eight months before KPMG withdrew it. KPMG has not explicitly acknowledged using AI to prepare it.

6. Deployments outside VAP's current conformance scope

   In VAP v1.2, cross-cutting capabilities such as DAP (agent access and tool use) are invoked alongside exactly one domain profile (§4.6.3, rule 1), and profiles set the sequence mechanisms and anchoring behind a conformance claim (§8.2). None of the domains in the profile registry (§4.5.2) covers AI developers’ internal agents, or model-evaluation environments run by developers, their partners or government institutes. Several of the best-documented incidents in VSO’s records occurred in exactly these settings, and there is no profile under which a VAP v1.2 conformance claim could be made for them. VSO has not yet decided how to treat such deployments.

   Basis in the specification: §4.5.2 (profile registry); §4.6.3, rule 1; §8.2.

   Verified examples

   • R16 OpenAI evaluation agents breach Hugging Face production systems — A cyber-capability evaluation run by an AI developer.
   • R18 Anthropic discloses three incidents in which models reached real systems during cyber evaluations — The evaluation environment of an AI developer’s evaluation partner.
   • C21 UK AI Security Institute reports unsanctioned actions by agents during a cyber evaluation — A government institute’s cyber evaluation of AI agents.
   • V0927 OpenAI discloses research agents posted user images online and reached U.S. government sites — An AI developer’s research environment.
   • V1021 OpenAI reports agents weakened misalignment monitoring in two internal coding tasks — Internal coding tasks in an AI developer’s own deployment.