documentation/docs/strategy/measurement.mdx

---
id: measurement
title: Skills Measurement
sidebar_label: Measurement
description: Which KPIs to collect during skills adoption, where the evidence comes from, how to calculate them and when to report.
hide_table_of_contents: true
---

import ArchifyDiagram from '@site/src/components/ArchifyDiagram';

<div className="strategy-content">

Measure adoption in stages: first establish today’s performance, then follow the pilot, extend coverage and keep skills current. Count use separately from outcomes—more use does not by itself mean better work.

<ArchifyDiagram name="strategy-measurement" title="Skills adoption measurement: baseline, pilot, broader adoption and monitoring" story />

**Recommended collection plan · Target operating model, not installed telemetry.** Confirm the proposed roles and reporting cadence below before collecting data.

## 1. Establish the baseline

Choose one workflow, such as reviewing a code change with a review skill. Agree what counts as a started task, independent acceptance and the follow-up period before counting outcomes. Collect current-practice evidence over a complete comparable workflow cycle and until the agreed sample requirement is met; use the same rules in the pilot.

Start with a controlled worksheet or export. Join task IDs to acceptance records, PR/commit IDs, CI results and later defects. Record the exact skill revision and client configuration for pilot work; keep missing versions unknown. Repository and CI evidence support verification, but a green build alone is not task acceptance.

## 2. Measure the pilot

Start with these four questions. Capture inputs as work happens; do not reconstruct review effort from elapsed PR time.

### Are tasks reaching acceptance?

**Collect:** task starts and independent acceptance, failure or abandonment records from the work tracker. **Calculate:** accepted tasks / all eligible starts whose observation period has ended × 100. Failed, abandoned and unresolved tasks stay in the denominator; show unknown outcomes and coverage separately. This is [validated task success](./reference/measurement-contracts.mdx#validated-task-success), not a count of agent completions. **Weekly:** Quality Engineering reviews the pilot cohort.

### Is quality holding up?

**Collect:** accepted change IDs, linked confirmed defects, severity and follow-up dates from review and defect records; retain PR/CI evidence. **Calculate:** changes with at least one confirmed defect / accepted changes with completed follow-up × 100. Count each affected change once, not each defect. Missing links do not mean clean work. **Monthly:** Quality Engineering reviews [escaped quality](./reference/measurement-contracts.mdx#changes-with-escaped-quality-issues) after follow-up, alongside severe incident counts.

### What does an accepted outcome cost?

**Collect:** invoices and usage costs plus sampled human review/rework minutes at approved rates, including unsuccessful attempts and allocated licenses/tooling. **Calculate:** total allocated workflow cost / independently accepted tasks in the same matured start cohort. Keep effort separate from waiting time. Missing buckets make this a partial estimate, not total cost. **Monthly:** Finance & Accounting reconciles [cost per accepted outcome](./reference/measurement-contracts.mdx#cost-per-accepted-outcome), restating late invoices.

### Can people confidently verify the work?

**Collect:** an anonymous 1–5 survey after a pilot workflow cycle: verification confidence, manageable workload and usefulness as separate items. **Calculate:** sum of valid confidence scores / valid responses; also show the distribution and responses / invitations × 100. This guides enablement, not claims of saved time. **After each pilot cycle:** People & Administration reviews [confidence and cognitive load](./reference/measurement-contracts.mdx#verification-confidence-and-cognitive-load); protect small groups.

### What a pilot report looks like

**Illustrative arithmetic, not company results or targets.** The bundled synthetic fixture shows four matured starts: one accepted, one failed, one abandoned and one unresolved. Confirmed success is **1 / 4 × 100 = 25%**, with **75%** known-outcome coverage. A newer start has not finished its observation period and is shown separately.

Recorded compute spend is **EUR 60 / 1 = EUR 60** per accepted outcome—a **partial estimate** because licenses, tooling, review and rework are missing. Five survey responses total 20 points: **20 / 5 = 4** out of 5; response rate is **5 / 10 × 100 = 50%**. The report calls for reconciling the unknown task and missing costs, not claiming a skills benefit. Reproduce it with `node scripts/ai-kpi.mjs`; [fixture and limits](./reference/measurement-contracts.mdx#run-the-synthetic-measurement-example).

## 3. Extend collection as adoption grows

Keep those outcome measures, then add reach. An adoption percentage is meaningless without a known eligible population.

**Are supported skills available?** Freeze the expected active installation inventory; compare observed bundle/adapter revisions with the approved support and revocation records. [Supported bundle coverage](./reference/measurement-contracts.mdx#supported-bundle-coverage) = fresh, verified supported installations / all expected active installations × 100. Offline or stale entries stay in the denominator. **Weekly and at revocation:** Platform & IT reconciles coverage.

**Are approved workflows being used?** Freeze the lawfully enrolled participant cohort; collect workflow-linked use of an approved revision through a qualified adapter or a separately labeled declaration. [Approved-workflow adoption](./reference/measurement-contracts.mdx#approved-workflow-adoption) = eligible participants with at least one observed approved use / eligible invited participants × 100. Report this as an observed lower bound with source coverage: missing use is unknown. Installation and loading are not use. **Monthly:** Data & AI reviews reach without invocation quotas or employee rankings.

## 4. Keep the collection running

**Are skills still maintained?** From the versioned catalog, join each active approved revision to its owner, substantive review evidence and due date. [Review coverage](./reference/measurement-contracts.mdx#skills-within-substantive-review-window) = revisions with all three current / all active approved revisions × 100. **Monthly and each release:** Data & AI reports overdue/ownerless revisions and deprecated or revoked versions still in use—not just a percentage.

At each report, compare like workflows with the baseline and publish the period, counts, exclusions and source coverage. Zero denominators mean “not available.” Trends are descriptive, not causal ROI. Act on critical policy effects immediately; add selection precision/recall at qualification and service-level delivery or departmental outcome measures when their source links are ready.

Agree purpose, lawful basis, access and retention first. No raw prompts, code or secrets by default; suppress people cohorts below five and prevent identifying drill-downs.

[Full KPI reference →](./reference/measurement-contracts.mdx) preserves all definitions, collection safeguards and the distinction between synthetic calculations and unimplemented metrics.

**Next:** [Maintain skills using the evidence →](./skills-maintenance.mdx)

</div>