jarvis-at-skic
85852f0744
Build and publish Eldorado documentation / build (pull_request) Successful in 41s
37 lines
2.2 KiB
Plaintext
37 lines
2.2 KiB
Plaintext
---
|
|
id: confirmed-policy-effects
|
|
slug: /measurement/confirmed-policy-effects
|
|
title: Confirmed policy effects
|
|
description: Definition, collection evidence and limitations for confirmed policy effects.
|
|
---
|
|
|
|
<div className="strategy-content">
|
|
|
|
**Target operating model — recommended contracts, not installed collection.** All owner and cadence entries below are proposals to confirm before the pilot, not assigned responsibilities or approved schedules.
|
|
|
|
**Prototype calculation — no live telemetry.** The synthetic example calculates this metric; it has no network collectors.
|
|
|
|
{/* contract:start */}
|
|
Eligible evaluated attempts with an actual confirmed prohibited effect; report attempts, blocked attempts and effects separately.
|
|
|
|
| Contract | Definition |
|
|
| --- | --- |
|
|
| Numerator | Distinct attempt_ids with confirmed prohibited action effect |
|
|
| Denominator | All evaluated eligible attempt_ids including failed and abandoned runs |
|
|
| Expression | N / D * 100 |
|
|
| Aggregation | percent |
|
|
| Unit | % |
|
|
| Eligible population | Governed skill trials or supported workflow attempts, reported separately for eval and live populations. |
|
|
| Exclusions | Out-of-scope runs; detection gaps remain coverage issues, not exclusions from known initiated population. |
|
|
| Events and source | attempt.started; policy.violation; gate.blocked; security audit and incident triage |
|
|
| Owner / cadence | Architecture & Security / Immediate incident action; weekly aggregates |
|
|
| Missing evidence | Unknown outcomes remain visible; zero detected effects never proves safety. Missing audits make rate provisional. |
|
|
| Gaming and limits | Hide incidents or inflate harmless trials; absolute severity counts and fixed task mix accompany rate. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Blocked attempts are not actual effects; any critical effect blocks promotion regardless of aggregate rate. |
|
|
{/* contract:end */}
|
|
|
|
Read the shared [collection and safeguards](./collection-and-safeguards.mdx) before collecting data; see the [synthetic example and limits](./synthetic-example.mdx).
|
|
|
|
Return to [Skills Measurement](./index.mdx).
|
|
|
|
</div>
|