docs: separate workflow effectiveness from skill health
Build and publish Eldorado documentation / build (pull_request) Successful in 52s
Build and publish Eldorado documentation / build (pull_request) Successful in 52s
This commit is contained in:
1 parent
386133c8b6
commit
e22d3a5c13
12 files changed
+142
-587
No files matched your search
@@ -3,7 +3,7 @@ id: index
|
||||
slug: /measurement
|
||||
title: Skills Measurement
|
||||
sidebar_label: Measurement
|
||||
description: Registry of available candidate KPIs and metrics, their applicability, evidence requirements, calculations and limitations.
|
||||
description: Optional contracts for workflow effectiveness and skills effectiveness and health, with distinct populations and honest implementation limits.
|
||||
pagination_prev: stage-01-skills-adoption
|
||||
pagination_next: baseline
|
||||
hide_table_of_contents: true
|
||||
@@ -13,15 +13,15 @@ import ArchifyDiagram from '@site/src/components/ArchifyDiagram';
|
||||
|
||||
<div className="strategy-content">
|
||||
|
||||
Browse available candidate KPIs and metrics. Registry inclusion does not require a metric to be used, collected or baselined. Count use separately from outcomes—more use does not by itself mean better work.
|
||||
Start with how work performs today; compare relevant outcomes after introducing skills. Separately assess whether the skills are effective, supported, used and maintained. Registry inclusion does not require a metric to be used, collected or baselined. More adoption does not prove improved work.
|
||||
|
||||
<ArchifyDiagram name="strategy-measurement" title="An optional metric registry: browse distinct questions and inspect each contract" story />
|
||||
<ArchifyDiagram name="strategy-measurement" title="Workflow effectiveness / Skills effectiveness & health" story />
|
||||
|
||||
**Target operating model — candidate definitions, not installed telemetry or a collection plan.** Proposed owners and cadences in the contracts apply only if selected and agreed for a workflow.
|
||||
|
||||
## Read a candidate contract
|
||||
|
||||
Each entry explains meaning and applicability, calculation, required inputs and evidence, exclusions, missing-evidence behavior, limitations and implementation status. Selection measures concern qualification; coverage and adoption concern reach; DORA measures concern service delivery. Outcome, safety, cost, experience and maintenance measures retain their distinct populations. A delivery metric need not fit a non-delivery workflow.
|
||||
Each entry explains meaning and applicability, calculation, required inputs and evidence, exclusions, missing-evidence behavior, limitations and implementation status. The two groups below classify every contract by its primary question; conditional reuse never merges populations. A delivery metric need not fit a non-delivery workflow.
|
||||
|
||||
To select a relevant subset and establish actual starting evidence, use [Baseline](../baseline.mdx). Pilot application belongs to [Adoption](../skills-adoption.mdx); ongoing interpretation belongs to [Maintenance](../skills-maintenance.mdx).
|
||||
|
||||
@@ -33,22 +33,37 @@ The nine calculations cover task success, selection precision and recall, change
|
||||
|
||||
Each contract is a prospective definition, not a rollout obligation. No live source or observed baseline is established. N and D refer to the numerator and denominator shown for that metric.
|
||||
|
||||
1. [Validated task success](./validated-task-success.mdx) — prototype calculation
|
||||
2. [Skill selection precision](./skill-selection-precision.mdx) — prototype calculation
|
||||
3. [Skill selection recall](./skill-selection-recall.mdx) — prototype calculation
|
||||
4. [Supported bundle coverage](./supported-bundle-coverage.mdx) — unimplemented metric
|
||||
5. [Approved-workflow adoption](./approved-workflow-adoption.mdx) — unimplemented metric
|
||||
6. [Change lead time](./change-lead-time.mdx) — unimplemented metric
|
||||
7. [Deployment frequency](./deployment-frequency.mdx) — unimplemented metric
|
||||
8. [Change fail rate](./change-fail-rate.mdx) — prototype calculation
|
||||
9. [Failed deployment recovery time](./failed-deployment-recovery-time.mdx) — unimplemented metric
|
||||
10. [Deployment rework rate](./deployment-rework-rate.mdx) — prototype calculation
|
||||
11. [Changes with escaped quality issues](./changes-with-escaped-quality-issues.mdx) — prototype calculation
|
||||
12. [Confirmed policy effects](./confirmed-policy-effects.mdx) — prototype calculation
|
||||
13. [Cost per accepted outcome](./cost-per-accepted-outcome.mdx) — prototype calculation
|
||||
14. [Verification confidence and cognitive load](./verification-confidence-and-cognitive-load.mdx) — prototype calculation
|
||||
15. [Skills within substantive review window](./skills-within-substantive-review-window.mdx) — unimplemented metric
|
||||
16. [Workflow-specific business acceptance](./workflow-specific-business-acceptance.mdx) — unimplemented metric
|
||||
### Workflow effectiveness
|
||||
|
||||
**Is the work getting better?** Baseline relevant current-workflow outcomes even before standardized skills exist, then compare the same outcomes during adoption and maintenance. Use comparable task mix, acceptance rules, follow-up periods and source coverage. Before/after trends alone are not causal evidence of a skills benefit.
|
||||
|
||||
1. [Validated task success](./validated-task-success.mdx) — **Prototype calculation.** Independently accepted outcomes among matured workflow starts; failures, abandonment and unresolved outcomes remain in the denominator.
|
||||
2. [Change lead time](./change-lead-time.mdx) — **Unimplemented metric.** Delivery time for production changes in one service, not general task duration.
|
||||
3. [Deployment frequency](./deployment-frequency.mdx) — **Unimplemented metric.** Production delivery frequency for one service; frequency alone is not value.
|
||||
4. [Change fail rate](./change-fail-rate.mdx) — **Prototype calculation.** Production deployments causing failures; requires joined change/incident evidence.
|
||||
5. [Failed deployment recovery time](./failed-deployment-recovery-time.mdx) — **Unimplemented metric.** Recovery from failed production deployments; conditional on eligible incidents.
|
||||
6. [Deployment rework rate](./deployment-rework-rate.mdx) — **Prototype calculation.** Unplanned corrective production deployments, not all forms of task rework.
|
||||
7. [Changes with escaped quality issues](./changes-with-escaped-quality-issues.mdx) — **Prototype calculation.** Accepted changes with confirmed downstream issues after follow-up.
|
||||
8. [Confirmed policy effects](./confirmed-policy-effects.mdx) — **Prototype calculation.** Safety of supported workflow attempts; governed skill trials are a separate evaluation population, never pooled with live work.
|
||||
9. [Cost per accepted outcome](./cost-per-accepted-outcome.mdx) — **Prototype calculation.** Cost of the matured workflow cohort, including unsuccessful attempts and recorded review/rework effort. Missing cost buckets remain partial.
|
||||
10. [Verification confidence and cognitive load](./verification-confidence-and-cognitive-load.mdx) — **Prototype calculation.** Anonymous cohort experience: confidence, workload and usefulness separately, not measured time savings.
|
||||
11. [Workflow-specific business acceptance](./workflow-specific-business-acceptance.mdx) — **Unimplemented metric.** Acceptance against an agreed departmental rubric; requires a defined workflow and independent evidence.
|
||||
|
||||
Human effort is recorded as review/rework inputs where the cost contract supports it; the experience survey describes perception. Neither elapsed delivery time nor token counts establish effort saved. If the current workflow lacks the event semantics required by a contract (for example, governed attempts), record it as not applicable or unavailable with a reason; do not relabel unrelated records to manufacture a baseline.
|
||||
|
||||
### Skills effectiveness & health
|
||||
|
||||
**Are the skills fit for use and kept fit?** Qualify named skills and versions, observe supported use during adoption, and review or retire them in maintenance. Without applicable skills, bundles or enrolled use, skill-specific measures may be **N/A, not zero**. Once a population exists, missing observations are unknown, not evidence of non-use or good health.
|
||||
|
||||
1. [Skill selection precision](./skill-selection-precision.mdx) — **Prototype calculation.** Did the evaluated selection choose an applicable skill? Requires labeled qualification cases.
|
||||
2. [Skill selection recall](./skill-selection-recall.mdx) — **Prototype calculation.** Were expected skills selected? Requires the same labeled qualification population.
|
||||
3. [Supported bundle coverage](./supported-bundle-coverage.mdx) — **Unimplemented metric.** Are expected active installations on supported, non-revoked bundles/adapters?
|
||||
4. [Approved-workflow adoption](./approved-workflow-adoption.mdx) — **Unimplemented metric.** Is the enrolled cohort using approved skills in eligible workflows? Installation/loading is not use.
|
||||
5. [Skills within substantive review window](./skills-within-substantive-review-window.mdx) — **Unimplemented metric.** Do active approved revisions have current owners and substantive review evidence?
|
||||
|
||||
**Task-success population matters.** The canonical validated-task-success contract above measures workflow outcomes, not an evaluation benchmark. A named skill/version can be examined as a predeclared live-workflow slice under that same contract, without counting retries or multiple skills as extra tasks or attributing causality. Sandbox/evaluation cases remain excluded: a named skill/version evaluation must be reported separately with its own declared cases, acceptance evidence and limits. It is not an additional implemented calculation or a second conflicting task-success formula. The canonical page explains this boundary. Likewise, keep policy-effect evaluation trials separate from live attempts.
|
||||
|
||||
Use both tracks together to decide whether to improve, retain or retire a skill. Higher selection scores, wider adoption or current reviews cannot substitute for accepted work, safety, cost and human experience.
|
||||
|
||||
Read [Collection and safeguards](./collection-and-safeguards.mdx) for shared interpretation, event and privacy rules. [Run the synthetic measurement example](./synthetic-example.mdx) for reproducible arithmetic and limits.
|
||||
|
||||
|
||||
Reference in new issue
Block a user