docs: [verified] separate Measurement registry and workflow Baseline
Build and publish Eldorado documentation / build (pull_request) Successful in 46s

This commit is contained in:
jarvis-at-skic committed 2026-10-07 13:33:07 +00:00
1 parent c4b4c38e62
commit abe00994a1
30 files changed
+961 -548

No files matched your search

@@ -2,12 +2,12 @@
id: collection-and-safeguards
slug: /measurement/collection-and-safeguards
title: Collection and safeguards
description: Collection rules, baseline interpretation, event contracts and privacy safeguards.
description: Shared interpretation rules, event contracts and privacy safeguards for selected metrics.
---
<div className="strategy-content">
**Target operating model — recommended contracts, not installed collection.** All owner and cadence entries below are proposals to confirm before the pilot, not assigned responsibilities or approved schedules.
**Target operating model — recommended contracts, not installed collection.** Registry inclusion does not require use, collection or baselining. All owner and cadence entries are proposals to confirm for selected metrics, not assigned responsibilities or approved schedules.
{/* preserved:start */}
Introduction depends on the [roadmap gates](../roadmap.mdx); these pages do not describe installed controls or measured company results.
@@ -20,11 +20,11 @@ Begin with task acceptance, repository, CI, deployment and incident systems beca
Validated success uses all eligible initiated tasks whose predeclared observation window has matured. Failed and abandoned tasks remain nonsuccess. Unresolved or unobserved outcomes remain in the denominator, with known-outcome coverage shown alongside the conservative confirmed-success share. They are not relabeled as observed failure. A task cancelled before start is excluded; cancellation after start remains visible. Count a task once across retries and multiple skills. For timeliness, also show unfinished work and age; accepted-only lead-time distributions have survivorship limitations.
Track all five service-level DORA measures: commit-to-production lead time, deployment frequency, change fail rate, failed deployment recovery time and deployment rework rate. Do not substitute PR-open-to-merge time or all-incident MTTR. Pair them with matured escaped-quality rates, policy effects, total cost per accepted outcome, support coverage, adoption, human experience and catalog maintenance. Keep customer/business outcomes linked to their departmental hypotheses; do not infer conversion, trust or revenue improvement from delivery telemetry.
The registry includes five service-level DORA measures: commit-to-production lead time, deployment frequency, change fail rate, failed deployment recovery time and deployment rework rate. Select them only where applicable; do not substitute PR-open-to-merge time or all-incident MTTR. Where relevant, interpret them alongside selected quality, policy, cost, support, adoption, experience or maintenance evidence. Keep customer/business outcomes linked to their departmental hypotheses; do not infer conversion, trust or revenue improvement from delivery telemetry.
### Baseline and interpretation
### Interpretation limits {#baseline-and-interpretation}
Before a pilot, freeze eligibility and acceptance rules and verify event joins. Collect prospective no-skill/current-practice evidence over a complete comparable workflow cycle and until the preregistered precision/sample requirement is met. Do not select a convenient end date after seeing results. Retain task class, service, skill, client/model configuration and concurrent changes. A matched or randomized comparison may be used when practical and ethical; label observational trends as descriptive and report confounding, selection and uncertainty.
See [Baseline](../baseline.mdx) for selecting relevant metrics and establishing comparable current-practice evidence. Pilot application belongs to [Adoption](../skills-adoption.mdx); ongoing reporting belongs to [Maintenance](../skills-maintenance.mdx).
Elapsed time and human effort are different quantities. Parallel agents may reduce waiting while increasing review or compute. Include failed attempts and human review/rework in cost, do not multiply a multi-skill task's value across every invocation, and do not convert survey perceptions into proven time savings. Small or incomplete populations yield evidence limits, not an ROI percentage.
+12 -54
View File
@@ -3,7 +3,9 @@ id: index
slug: /measurement
title: Skills Measurement
sidebar_label: Measurement
description: Which KPIs to collect during skills adoption, where the evidence comes from, how to calculate them and when to report.
description: Registry of available candidate KPIs and metrics, their applicability, evidence requirements, calculations and limitations.
pagination_prev: stage-01-skills-adoption
pagination_next: baseline
hide_table_of_contents: true
---
@@ -11,59 +13,17 @@ import ArchifyDiagram from '@site/src/components/ArchifyDiagram';
<div className="strategy-content">
Measure adoption in stages: first establish today’s performance, then follow the pilot, extend coverage and keep skills current. Count use separately from outcomes—more use does not by itself mean better work.
Browse available candidate KPIs and metrics. Registry inclusion does not require a metric to be used, collected or baselined. Count use separately from outcomes—more use does not by itself mean better work.
<ArchifyDiagram name="strategy-measurement" title="Skills adoption measurement: baseline, pilot, broader adoption and monitoring" story />
<ArchifyDiagram name="strategy-measurement" title="An optional metric registry: browse distinct questions and inspect each contract" story />
**Recommended collection plan · Target operating model, not installed telemetry.** Confirm the proposed roles and reporting cadence below before collecting data.
**Target operating model — candidate definitions, not installed telemetry or a collection plan.** Proposed owners and cadences in the contracts apply only if selected and agreed for a workflow.
## 1. Establish the baseline
## Read a candidate contract
Choose one workflow, such as reviewing a code change with a review skill. Agree what counts as a started task, independent acceptance and the follow-up period before counting outcomes. Collect current-practice evidence over a complete comparable workflow cycle and until the agreed sample requirement is met; use the same rules in the pilot.
Each entry explains meaning and applicability, calculation, required inputs and evidence, exclusions, missing-evidence behavior, limitations and implementation status. Selection measures concern qualification; coverage and adoption concern reach; DORA measures concern service delivery. Outcome, safety, cost, experience and maintenance measures retain their distinct populations. A delivery metric need not fit a non-delivery workflow.
Start with a controlled worksheet or export. Join task IDs to acceptance records, PR/commit IDs, CI results and later defects. Record the exact skill revision and client configuration for pilot work; keep missing versions unknown. Repository and CI evidence support verification, but a green build alone is not task acceptance.
## 2. Measure the pilot
Start with these four questions. Capture inputs as work happens; do not reconstruct review effort from elapsed PR time.
### Are tasks reaching acceptance?
**Collect:** task starts and independent acceptance, failure or abandonment records from the work tracker. **Calculate:** accepted tasks / all eligible starts whose observation period has ended × 100. Failed, abandoned and unresolved tasks stay in the denominator; show unknown outcomes and coverage separately. This is [validated task success](./validated-task-success.mdx), not a count of agent completions. **Weekly:** Quality Engineering reviews the pilot cohort.
### Is quality holding up?
**Collect:** accepted change IDs, linked confirmed defects, severity and follow-up dates from review and defect records; retain PR/CI evidence. **Calculate:** changes with at least one confirmed defect / accepted changes with completed follow-up × 100. Count each affected change once, not each defect. Missing links do not mean clean work. **Monthly:** Quality Engineering reviews [escaped quality](./changes-with-escaped-quality-issues.mdx) after follow-up, alongside severe incident counts.
### What does an accepted outcome cost?
**Collect:** invoices and usage costs plus sampled human review/rework minutes at approved rates, including unsuccessful attempts and allocated licenses/tooling. **Calculate:** total allocated workflow cost / independently accepted tasks in the same matured start cohort. Keep effort separate from waiting time. Missing buckets make this a partial estimate, not total cost. **Monthly:** Finance & Accounting reconciles [cost per accepted outcome](./cost-per-accepted-outcome.mdx), restating late invoices.
### Can people confidently verify the work?
**Collect:** an anonymous 1–5 survey after a pilot workflow cycle: verification confidence, manageable workload and usefulness as separate items. **Calculate:** sum of valid confidence scores / valid responses; also show the distribution and responses / invitations × 100. This guides enablement, not claims of saved time. **After each pilot cycle:** People & Administration reviews [confidence and cognitive load](./verification-confidence-and-cognitive-load.mdx); protect small groups.
### What a pilot report looks like
**Illustrative arithmetic, not company results or targets.** The bundled synthetic fixture shows four matured starts: one accepted, one failed, one abandoned and one unresolved. Confirmed success is **1 / 4 × 100 = 25%**, with **75%** known-outcome coverage. A newer start has not finished its observation period and is shown separately.
Recorded compute spend is **EUR 60 / 1 = EUR 60** per accepted outcome—a **partial estimate** because licenses, tooling, review and rework are missing. Five survey responses total 20 points: **20 / 5 = 4** out of 5; response rate is **5 / 10 × 100 = 50%**. The report calls for reconciling the unknown task and missing costs, not claiming a skills benefit. Reproduce it with `node scripts/ai-kpi.mjs`; [fixture and limits](./synthetic-example.mdx).
## 3. Extend collection as adoption grows
Keep those outcome measures, then add reach. An adoption percentage is meaningless without a known eligible population.
**Are supported skills available?** Freeze the expected active installation inventory; compare observed bundle/adapter revisions with the approved support and revocation records. [Supported bundle coverage](./supported-bundle-coverage.mdx) = fresh, verified supported installations / all expected active installations × 100. Offline or stale entries stay in the denominator. **Weekly and at revocation:** Platform & IT reconciles coverage.
**Are approved workflows being used?** Freeze the lawfully enrolled participant cohort; collect workflow-linked use of an approved revision through a qualified adapter or a separately labeled declaration. [Approved-workflow adoption](./approved-workflow-adoption.mdx) = eligible participants with at least one observed approved use / eligible invited participants × 100. Report this as an observed lower bound with source coverage: missing use is unknown. Installation and loading are not use. **Monthly:** Data & AI reviews reach without invocation quotas or employee rankings.
## 4. Keep the collection running
**Are skills still maintained?** From the versioned catalog, join each active approved revision to its owner, substantive review evidence and due date. [Review coverage](./skills-within-substantive-review-window.mdx) = revisions with all three current / all active approved revisions × 100. **Monthly and each release:** Data & AI reports overdue/ownerless revisions and deprecated or revoked versions still in use—not just a percentage.
At each report, compare like workflows with the baseline and publish the period, counts, exclusions and source coverage. Zero denominators mean “not available.” Trends are descriptive, not causal ROI. Act on critical policy effects immediately; add selection precision/recall at qualification and service-level delivery or departmental outcome measures when their source links are ready.
Agree purpose, lawful basis, access and retention first. No raw prompts, code or secrets by default; suppress people cohorts below five and prevent identifying drill-downs.
To select a relevant subset and establish actual starting evidence, use [Baseline](../baseline.mdx). Pilot application belongs to [Adoption](../skills-adoption.mdx); ongoing interpretation belongs to [Maintenance](../skills-maintenance.mdx).
## Metric definitions
@@ -71,9 +31,7 @@ There are **16 contracts: 9 prototype calculations and 7 unimplemented metrics**
The nine calculations cover task success, selection precision and recall, change fail rate, deployment rework, escaped quality, policy effects, cost and the primary human-experience item. The seven omissions are supported bundle coverage, approved-workflow adoption, change lead time, deployment frequency, failed deployment recovery time, substantive-review coverage and workflow-specific business acceptance.
Each contract is a prospective collection design. No live source or observed baseline is established. N and D refer to the numerator and denominator shown for that metric.
Selection measures concern qualification; coverage and adoption concern reach; DORA measures concern service delivery. Outcome, safety, cost, experience and maintenance measures retain their distinct populations.
Each contract is a prospective definition, not a rollout obligation. No live source or observed baseline is established. N and D refer to the numerator and denominator shown for that metric.
1. [Validated task success](./validated-task-success.mdx) — prototype calculation
2. [Skill selection precision](./skill-selection-precision.mdx) — prototype calculation
@@ -92,8 +50,8 @@ Selection measures concern qualification; coverage and adoption concern reach; D
15. [Skills within substantive review window](./skills-within-substantive-review-window.mdx) — unimplemented metric
16. [Workflow-specific business acceptance](./workflow-specific-business-acceptance.mdx) — unimplemented metric
Read [Collection and safeguards](./collection-and-safeguards.mdx) for shared baseline, event and privacy rules. [Run the synthetic measurement example](./synthetic-example.mdx) for reproducible arithmetic and limits.
Read [Collection and safeguards](./collection-and-safeguards.mdx) for shared interpretation, event and privacy rules. [Run the synthetic measurement example](./synthetic-example.mdx) for reproducible arithmetic and limits.
**Next:** [Maintain skills using the evidence →](../skills-maintenance.mdx)
**Next:** [Establish a relevant workflow baseline →](../baseline.mdx)
</div>