Build and publish Eldorado documentation / build (pull_request) Successful in 52s
73 lines
7.8 KiB
Plaintext
73 lines
7.8 KiB
Plaintext
---
|
|
id: index
|
|
slug: /measurement
|
|
title: Skills Measurement
|
|
sidebar_label: Measurement
|
|
description: Optional contracts for workflow effectiveness and skills effectiveness and health, with distinct populations and honest implementation limits.
|
|
pagination_prev: stage-01-skills-adoption
|
|
pagination_next: baseline
|
|
hide_table_of_contents: true
|
|
---
|
|
|
|
import ArchifyDiagram from '@site/src/components/ArchifyDiagram';
|
|
|
|
<div className="strategy-content">
|
|
|
|
Start with how work performs today; compare relevant outcomes after introducing skills. Separately assess whether the skills are effective, supported, used and maintained. Registry inclusion does not require a metric to be used, collected or baselined. More adoption does not prove improved work.
|
|
|
|
<ArchifyDiagram name="strategy-measurement" title="Workflow effectiveness / Skills effectiveness & health" story />
|
|
|
|
**Target operating model — candidate definitions, not installed telemetry or a collection plan.** Proposed owners and cadences in the contracts apply only if selected and agreed for a workflow.
|
|
|
|
## Read a candidate contract
|
|
|
|
Each entry explains meaning and applicability, calculation, required inputs and evidence, exclusions, missing-evidence behavior, limitations and implementation status. The two groups below classify every contract by its primary question; conditional reuse never merges populations. A delivery metric need not fit a non-delivery workflow.
|
|
|
|
To select a relevant subset and establish actual starting evidence, use [Baseline](../baseline.mdx). Pilot application belongs to [Adoption](../skills-adoption.mdx); ongoing interpretation belongs to [Maintenance](../skills-maintenance.mdx).
|
|
|
|
## Metric definitions
|
|
|
|
There are **16 contracts: 9 prototype calculations and 7 unimplemented metrics**. The prototype has no network collectors. None of these counts establishes live telemetry.
|
|
|
|
The nine calculations cover task success, selection precision and recall, change fail rate, deployment rework, escaped quality, policy effects, cost and the primary human-experience item. The seven omissions are supported bundle coverage, approved-workflow adoption, change lead time, deployment frequency, failed deployment recovery time, substantive-review coverage and workflow-specific business acceptance.
|
|
|
|
Each contract is a prospective definition, not a rollout obligation. No live source or observed baseline is established. N and D refer to the numerator and denominator shown for that metric.
|
|
|
|
### Workflow effectiveness
|
|
|
|
**Is the work getting better?** Baseline relevant current-workflow outcomes even before standardized skills exist, then compare the same outcomes during adoption and maintenance. Use comparable task mix, acceptance rules, follow-up periods and source coverage. Before/after trends alone are not causal evidence of a skills benefit.
|
|
|
|
1. [Validated task success](./validated-task-success.mdx) — **Prototype calculation.** Independently accepted outcomes among matured workflow starts; failures, abandonment and unresolved outcomes remain in the denominator.
|
|
2. [Change lead time](./change-lead-time.mdx) — **Unimplemented metric.** Delivery time for production changes in one service, not general task duration.
|
|
3. [Deployment frequency](./deployment-frequency.mdx) — **Unimplemented metric.** Production delivery frequency for one service; frequency alone is not value.
|
|
4. [Change fail rate](./change-fail-rate.mdx) — **Prototype calculation.** Production deployments causing failures; requires joined change/incident evidence.
|
|
5. [Failed deployment recovery time](./failed-deployment-recovery-time.mdx) — **Unimplemented metric.** Recovery from failed production deployments; conditional on eligible incidents.
|
|
6. [Deployment rework rate](./deployment-rework-rate.mdx) — **Prototype calculation.** Unplanned corrective production deployments, not all forms of task rework.
|
|
7. [Changes with escaped quality issues](./changes-with-escaped-quality-issues.mdx) — **Prototype calculation.** Accepted changes with confirmed downstream issues after follow-up.
|
|
8. [Confirmed policy effects](./confirmed-policy-effects.mdx) — **Prototype calculation.** Safety of supported workflow attempts; governed skill trials are a separate evaluation population, never pooled with live work.
|
|
9. [Cost per accepted outcome](./cost-per-accepted-outcome.mdx) — **Prototype calculation.** Cost of the matured workflow cohort, including unsuccessful attempts and recorded review/rework effort. Missing cost buckets remain partial.
|
|
10. [Verification confidence and cognitive load](./verification-confidence-and-cognitive-load.mdx) — **Prototype calculation.** Anonymous cohort experience: confidence, workload and usefulness separately, not measured time savings.
|
|
11. [Workflow-specific business acceptance](./workflow-specific-business-acceptance.mdx) — **Unimplemented metric.** Acceptance against an agreed departmental rubric; requires a defined workflow and independent evidence.
|
|
|
|
Human effort is recorded as review/rework inputs where the cost contract supports it; the experience survey describes perception. Neither elapsed delivery time nor token counts establish effort saved. If the current workflow lacks the event semantics required by a contract (for example, governed attempts), record it as not applicable or unavailable with a reason; do not relabel unrelated records to manufacture a baseline.
|
|
|
|
### Skills effectiveness & health
|
|
|
|
**Are the skills fit for use and kept fit?** Qualify named skills and versions, observe supported use during adoption, and review or retire them in maintenance. Without applicable skills, bundles or enrolled use, skill-specific measures may be **N/A, not zero**. Once a population exists, missing observations are unknown, not evidence of non-use or good health.
|
|
|
|
1. [Skill selection precision](./skill-selection-precision.mdx) — **Prototype calculation.** Did the evaluated selection choose an applicable skill? Requires labeled qualification cases.
|
|
2. [Skill selection recall](./skill-selection-recall.mdx) — **Prototype calculation.** Were expected skills selected? Requires the same labeled qualification population.
|
|
3. [Supported bundle coverage](./supported-bundle-coverage.mdx) — **Unimplemented metric.** Are expected active installations on supported, non-revoked bundles/adapters?
|
|
4. [Approved-workflow adoption](./approved-workflow-adoption.mdx) — **Unimplemented metric.** Is the enrolled cohort using approved skills in eligible workflows? Installation/loading is not use.
|
|
5. [Skills within substantive review window](./skills-within-substantive-review-window.mdx) — **Unimplemented metric.** Do active approved revisions have current owners and substantive review evidence?
|
|
|
|
**Task-success population matters.** The canonical validated-task-success contract above measures workflow outcomes, not an evaluation benchmark. A named skill/version can be examined as a predeclared live-workflow slice under that same contract, without counting retries or multiple skills as extra tasks or attributing causality. Sandbox/evaluation cases remain excluded: a named skill/version evaluation must be reported separately with its own declared cases, acceptance evidence and limits. It is not an additional implemented calculation or a second conflicting task-success formula. The canonical page explains this boundary. Likewise, keep policy-effect evaluation trials separate from live attempts.
|
|
|
|
Use both tracks together to decide whether to improve, retain or retire a skill. Higher selection scores, wider adoption or current reviews cannot substitute for accepted work, safety, cost and human experience.
|
|
|
|
Read [Collection and safeguards](./collection-and-safeguards.mdx) for shared interpretation, event and privacy rules. [Run the synthetic measurement example](./synthetic-example.mdx) for reproducible arithmetic and limits.
|
|
|
|
**Next:** [Establish a relevant workflow baseline →](../baseline.mdx)
|
|
|
|
</div>
|