docs: separate workflow effectiveness from skill health
Build and publish Eldorado documentation / build (pull_request) Successful in 52s

This commit is contained in:
jarvis-at-skic committed 2026-10-08 11:05:00 +00:00
1 parent 386133c8b6
commit e22d3a5c13
12 files changed
+142 -587

No files matched your search

+35 -20
View File
@@ -3,7 +3,7 @@ id: index
slug: /measurement
title: Skills Measurement
sidebar_label: Measurement
description: Registry of available candidate KPIs and metrics, their applicability, evidence requirements, calculations and limitations.
description: Optional contracts for workflow effectiveness and skills effectiveness and health, with distinct populations and honest implementation limits.
pagination_prev: stage-01-skills-adoption
pagination_next: baseline
hide_table_of_contents: true
@@ -13,15 +13,15 @@ import ArchifyDiagram from '@site/src/components/ArchifyDiagram';
<div className="strategy-content">
Browse available candidate KPIs and metrics. Registry inclusion does not require a metric to be used, collected or baselined. Count use separately from outcomes—more use does not by itself mean better work.
Start with how work performs today; compare relevant outcomes after introducing skills. Separately assess whether the skills are effective, supported, used and maintained. Registry inclusion does not require a metric to be used, collected or baselined. More adoption does not prove improved work.
<ArchifyDiagram name="strategy-measurement" title="An optional metric registry: browse distinct questions and inspect each contract" story />
<ArchifyDiagram name="strategy-measurement" title="Workflow effectiveness / Skills effectiveness & health" story />
**Target operating model — candidate definitions, not installed telemetry or a collection plan.** Proposed owners and cadences in the contracts apply only if selected and agreed for a workflow.
## Read a candidate contract
Each entry explains meaning and applicability, calculation, required inputs and evidence, exclusions, missing-evidence behavior, limitations and implementation status. Selection measures concern qualification; coverage and adoption concern reach; DORA measures concern service delivery. Outcome, safety, cost, experience and maintenance measures retain their distinct populations. A delivery metric need not fit a non-delivery workflow.
Each entry explains meaning and applicability, calculation, required inputs and evidence, exclusions, missing-evidence behavior, limitations and implementation status. The two groups below classify every contract by its primary question; conditional reuse never merges populations. A delivery metric need not fit a non-delivery workflow.
To select a relevant subset and establish actual starting evidence, use [Baseline](../baseline.mdx). Pilot application belongs to [Adoption](../skills-adoption.mdx); ongoing interpretation belongs to [Maintenance](../skills-maintenance.mdx).
@@ -33,22 +33,37 @@ The nine calculations cover task success, selection precision and recall, change
Each contract is a prospective definition, not a rollout obligation. No live source or observed baseline is established. N and D refer to the numerator and denominator shown for that metric.
1. [Validated task success](./validated-task-success.mdx) — prototype calculation
2. [Skill selection precision](./skill-selection-precision.mdx) — prototype calculation
3. [Skill selection recall](./skill-selection-recall.mdx) — prototype calculation
4. [Supported bundle coverage](./supported-bundle-coverage.mdx) — unimplemented metric
5. [Approved-workflow adoption](./approved-workflow-adoption.mdx) — unimplemented metric
6. [Change lead time](./change-lead-time.mdx) — unimplemented metric
7. [Deployment frequency](./deployment-frequency.mdx) — unimplemented metric
8. [Change fail rate](./change-fail-rate.mdx) — prototype calculation
9. [Failed deployment recovery time](./failed-deployment-recovery-time.mdx) — unimplemented metric
10. [Deployment rework rate](./deployment-rework-rate.mdx) — prototype calculation
11. [Changes with escaped quality issues](./changes-with-escaped-quality-issues.mdx) — prototype calculation
12. [Confirmed policy effects](./confirmed-policy-effects.mdx) — prototype calculation
13. [Cost per accepted outcome](./cost-per-accepted-outcome.mdx) — prototype calculation
14. [Verification confidence and cognitive load](./verification-confidence-and-cognitive-load.mdx) — prototype calculation
15. [Skills within substantive review window](./skills-within-substantive-review-window.mdx) — unimplemented metric
16. [Workflow-specific business acceptance](./workflow-specific-business-acceptance.mdx) — unimplemented metric
### Workflow effectiveness
**Is the work getting better?** Baseline relevant current-workflow outcomes even before standardized skills exist, then compare the same outcomes during adoption and maintenance. Use comparable task mix, acceptance rules, follow-up periods and source coverage. Before/after trends alone are not causal evidence of a skills benefit.
1. [Validated task success](./validated-task-success.mdx) — **Prototype calculation.** Independently accepted outcomes among matured workflow starts; failures, abandonment and unresolved outcomes remain in the denominator.
2. [Change lead time](./change-lead-time.mdx) — **Unimplemented metric.** Delivery time for production changes in one service, not general task duration.
3. [Deployment frequency](./deployment-frequency.mdx) — **Unimplemented metric.** Production delivery frequency for one service; frequency alone is not value.
4. [Change fail rate](./change-fail-rate.mdx) — **Prototype calculation.** Production deployments causing failures; requires joined change/incident evidence.
5. [Failed deployment recovery time](./failed-deployment-recovery-time.mdx) — **Unimplemented metric.** Recovery from failed production deployments; conditional on eligible incidents.
6. [Deployment rework rate](./deployment-rework-rate.mdx) — **Prototype calculation.** Unplanned corrective production deployments, not all forms of task rework.
7. [Changes with escaped quality issues](./changes-with-escaped-quality-issues.mdx) — **Prototype calculation.** Accepted changes with confirmed downstream issues after follow-up.
8. [Confirmed policy effects](./confirmed-policy-effects.mdx) — **Prototype calculation.** Safety of supported workflow attempts; governed skill trials are a separate evaluation population, never pooled with live work.
9. [Cost per accepted outcome](./cost-per-accepted-outcome.mdx) — **Prototype calculation.** Cost of the matured workflow cohort, including unsuccessful attempts and recorded review/rework effort. Missing cost buckets remain partial.
10. [Verification confidence and cognitive load](./verification-confidence-and-cognitive-load.mdx) — **Prototype calculation.** Anonymous cohort experience: confidence, workload and usefulness separately, not measured time savings.
11. [Workflow-specific business acceptance](./workflow-specific-business-acceptance.mdx) — **Unimplemented metric.** Acceptance against an agreed departmental rubric; requires a defined workflow and independent evidence.
Human effort is recorded as review/rework inputs where the cost contract supports it; the experience survey describes perception. Neither elapsed delivery time nor token counts establish effort saved. If the current workflow lacks the event semantics required by a contract (for example, governed attempts), record it as not applicable or unavailable with a reason; do not relabel unrelated records to manufacture a baseline.
### Skills effectiveness & health
**Are the skills fit for use and kept fit?** Qualify named skills and versions, observe supported use during adoption, and review or retire them in maintenance. Without applicable skills, bundles or enrolled use, skill-specific measures may be **N/A, not zero**. Once a population exists, missing observations are unknown, not evidence of non-use or good health.
1. [Skill selection precision](./skill-selection-precision.mdx) — **Prototype calculation.** Did the evaluated selection choose an applicable skill? Requires labeled qualification cases.
2. [Skill selection recall](./skill-selection-recall.mdx) — **Prototype calculation.** Were expected skills selected? Requires the same labeled qualification population.
3. [Supported bundle coverage](./supported-bundle-coverage.mdx) — **Unimplemented metric.** Are expected active installations on supported, non-revoked bundles/adapters?
4. [Approved-workflow adoption](./approved-workflow-adoption.mdx) — **Unimplemented metric.** Is the enrolled cohort using approved skills in eligible workflows? Installation/loading is not use.
5. [Skills within substantive review window](./skills-within-substantive-review-window.mdx) — **Unimplemented metric.** Do active approved revisions have current owners and substantive review evidence?
**Task-success population matters.** The canonical validated-task-success contract above measures workflow outcomes, not an evaluation benchmark. A named skill/version can be examined as a predeclared live-workflow slice under that same contract, without counting retries or multiple skills as extra tasks or attributing causality. Sandbox/evaluation cases remain excluded: a named skill/version evaluation must be reported separately with its own declared cases, acceptance evidence and limits. It is not an additional implemented calculation or a second conflicting task-success formula. The canonical page explains this boundary. Likewise, keep policy-effect evaluation trials separate from live attempts.
Use both tracks together to decide whether to improve, retain or retire a skill. Higher selection scores, wider adoption or current reviews cannot substitute for accepted work, safety, cost and human experience.
Read [Collection and safeguards](./collection-and-safeguards.mdx) for shared interpretation, event and privacy rules. [Run the synthetic measurement example](./synthetic-example.mdx) for reproducible arithmetic and limits.