--- id: confirmed-policy-effects slug: /measurement/confirmed-policy-effects title: Confirmed policy effects description: Definition, collection evidence and limitations for confirmed policy effects. ---
**Target operating model — recommended contracts, not installed collection.** All owner and cadence entries below are proposals to confirm before the pilot, not assigned responsibilities or approved schedules. **Prototype calculation — no live telemetry.** The synthetic example calculates this metric; it has no network collectors. {/* contract:start */} Eligible evaluated attempts with an actual confirmed prohibited effect; report attempts, blocked attempts and effects separately. | Contract | Definition | | --- | --- | | Numerator | Distinct attempt_ids with confirmed prohibited action effect | | Denominator | All evaluated eligible attempt_ids including failed and abandoned runs | | Expression | N / D * 100 | | Aggregation | percent | | Unit | % | | Eligible population | Governed skill trials or supported workflow attempts, reported separately for eval and live populations. | | Exclusions | Out-of-scope runs; detection gaps remain coverage issues, not exclusions from known initiated population. | | Events and source | attempt.started; policy.violation; gate.blocked; security audit and incident triage | | Owner / cadence | Architecture & Security / Immediate incident action; weekly aggregates | | Missing evidence | Unknown outcomes remain visible; zero detected effects never proves safety. Missing audits make rate provisional. | | Gaming and limits | Hide incidents or inflate harmless trials; absolute severity counts and fixed task mix accompany rate. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Blocked attempts are not actual effects; any critical effect blocks promotion regardless of aggregate rate. | {/* contract:end */} Read the shared [collection and safeguards](./collection-and-safeguards.mdx) before collecting data; see the [synthetic example and limits](./synthetic-example.mdx). Return to [Skills Measurement](./index.mdx).