docs: separate workflow effectiveness from skill health
Build and publish Eldorado documentation / build (pull_request) Successful in 52s
Build and publish Eldorado documentation / build (pull_request) Successful in 52s
This commit is contained in:
1 parent
386133c8b6
commit
e22d3a5c13
12 files changed
+142
-587
No files matched your search
@@ -24,7 +24,9 @@ Describe the task, team or service, intended outcome, current tools, handoffs, c
|
||||
|
||||
## Select relevant metrics, with reasons
|
||||
|
||||
Choose the registry entries that answer this workflow's questions. Record why each selected metric applies, what decision it can inform, and why other candidates are irrelevant or deferred. There is no required universal subset. A service-level DORA metric, a qualification test and a people survey have different populations; they are not interchangeable.
|
||||
Start with **Workflow effectiveness**: select relevant current-workflow outcomes before standardized skills adoption—accepted work, delivery time, failures/rework, cost, human effort/experience or business acceptance where the contracts apply. Choose the registry entries that answer this workflow's questions. Record why each selected metric applies, what decision it can inform, and why other candidates are irrelevant or deferred. There is no required universal subset. A service-level DORA metric, a qualification test and a people survey have different populations; they are not interchangeable.
|
||||
|
||||
Keep **Skills effectiveness & health** separate. If no applicable skills exist, selection, bundle, adoption and review measures may be N/A, not zero; they are not prerequisites for baselining current work. If skills already exist, record that starting context and qualify named revisions separately. Do not mix sandbox skill evaluations into validated workflow task success or live policy-effect rates.
|
||||
|
||||
Agree what counts as a started task, independent acceptance and completed follow-up before counting outcomes. Freeze eligibility and acceptance rules and verify event joins. Select a complete comparable workflow cycle and a predeclared precision/sample requirement appropriate to the decision. Do not select a convenient end date after seeing results. Where those requirements cannot be met, record the resulting evidence limitation.
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@ id: index
|
||||
slug: /measurement
|
||||
title: Skills Measurement
|
||||
sidebar_label: Measurement
|
||||
description: Registry of available candidate KPIs and metrics, their applicability, evidence requirements, calculations and limitations.
|
||||
description: Optional contracts for workflow effectiveness and skills effectiveness and health, with distinct populations and honest implementation limits.
|
||||
pagination_prev: stage-01-skills-adoption
|
||||
pagination_next: baseline
|
||||
hide_table_of_contents: true
|
||||
@@ -13,15 +13,15 @@ import ArchifyDiagram from '@site/src/components/ArchifyDiagram';
|
||||
|
||||
<div className="strategy-content">
|
||||
|
||||
Browse available candidate KPIs and metrics. Registry inclusion does not require a metric to be used, collected or baselined. Count use separately from outcomes—more use does not by itself mean better work.
|
||||
Start with how work performs today; compare relevant outcomes after introducing skills. Separately assess whether the skills are effective, supported, used and maintained. Registry inclusion does not require a metric to be used, collected or baselined. More adoption does not prove improved work.
|
||||
|
||||
<ArchifyDiagram name="strategy-measurement" title="An optional metric registry: browse distinct questions and inspect each contract" story />
|
||||
<ArchifyDiagram name="strategy-measurement" title="Workflow effectiveness / Skills effectiveness & health" story />
|
||||
|
||||
**Target operating model — candidate definitions, not installed telemetry or a collection plan.** Proposed owners and cadences in the contracts apply only if selected and agreed for a workflow.
|
||||
|
||||
## Read a candidate contract
|
||||
|
||||
Each entry explains meaning and applicability, calculation, required inputs and evidence, exclusions, missing-evidence behavior, limitations and implementation status. Selection measures concern qualification; coverage and adoption concern reach; DORA measures concern service delivery. Outcome, safety, cost, experience and maintenance measures retain their distinct populations. A delivery metric need not fit a non-delivery workflow.
|
||||
Each entry explains meaning and applicability, calculation, required inputs and evidence, exclusions, missing-evidence behavior, limitations and implementation status. The two groups below classify every contract by its primary question; conditional reuse never merges populations. A delivery metric need not fit a non-delivery workflow.
|
||||
|
||||
To select a relevant subset and establish actual starting evidence, use [Baseline](../baseline.mdx). Pilot application belongs to [Adoption](../skills-adoption.mdx); ongoing interpretation belongs to [Maintenance](../skills-maintenance.mdx).
|
||||
|
||||
@@ -33,22 +33,37 @@ The nine calculations cover task success, selection precision and recall, change
|
||||
|
||||
Each contract is a prospective definition, not a rollout obligation. No live source or observed baseline is established. N and D refer to the numerator and denominator shown for that metric.
|
||||
|
||||
1. [Validated task success](./validated-task-success.mdx) — prototype calculation
|
||||
2. [Skill selection precision](./skill-selection-precision.mdx) — prototype calculation
|
||||
3. [Skill selection recall](./skill-selection-recall.mdx) — prototype calculation
|
||||
4. [Supported bundle coverage](./supported-bundle-coverage.mdx) — unimplemented metric
|
||||
5. [Approved-workflow adoption](./approved-workflow-adoption.mdx) — unimplemented metric
|
||||
6. [Change lead time](./change-lead-time.mdx) — unimplemented metric
|
||||
7. [Deployment frequency](./deployment-frequency.mdx) — unimplemented metric
|
||||
8. [Change fail rate](./change-fail-rate.mdx) — prototype calculation
|
||||
9. [Failed deployment recovery time](./failed-deployment-recovery-time.mdx) — unimplemented metric
|
||||
10. [Deployment rework rate](./deployment-rework-rate.mdx) — prototype calculation
|
||||
11. [Changes with escaped quality issues](./changes-with-escaped-quality-issues.mdx) — prototype calculation
|
||||
12. [Confirmed policy effects](./confirmed-policy-effects.mdx) — prototype calculation
|
||||
13. [Cost per accepted outcome](./cost-per-accepted-outcome.mdx) — prototype calculation
|
||||
14. [Verification confidence and cognitive load](./verification-confidence-and-cognitive-load.mdx) — prototype calculation
|
||||
15. [Skills within substantive review window](./skills-within-substantive-review-window.mdx) — unimplemented metric
|
||||
16. [Workflow-specific business acceptance](./workflow-specific-business-acceptance.mdx) — unimplemented metric
|
||||
### Workflow effectiveness
|
||||
|
||||
**Is the work getting better?** Baseline relevant current-workflow outcomes even before standardized skills exist, then compare the same outcomes during adoption and maintenance. Use comparable task mix, acceptance rules, follow-up periods and source coverage. Before/after trends alone are not causal evidence of a skills benefit.
|
||||
|
||||
1. [Validated task success](./validated-task-success.mdx) — **Prototype calculation.** Independently accepted outcomes among matured workflow starts; failures, abandonment and unresolved outcomes remain in the denominator.
|
||||
2. [Change lead time](./change-lead-time.mdx) — **Unimplemented metric.** Delivery time for production changes in one service, not general task duration.
|
||||
3. [Deployment frequency](./deployment-frequency.mdx) — **Unimplemented metric.** Production delivery frequency for one service; frequency alone is not value.
|
||||
4. [Change fail rate](./change-fail-rate.mdx) — **Prototype calculation.** Production deployments causing failures; requires joined change/incident evidence.
|
||||
5. [Failed deployment recovery time](./failed-deployment-recovery-time.mdx) — **Unimplemented metric.** Recovery from failed production deployments; conditional on eligible incidents.
|
||||
6. [Deployment rework rate](./deployment-rework-rate.mdx) — **Prototype calculation.** Unplanned corrective production deployments, not all forms of task rework.
|
||||
7. [Changes with escaped quality issues](./changes-with-escaped-quality-issues.mdx) — **Prototype calculation.** Accepted changes with confirmed downstream issues after follow-up.
|
||||
8. [Confirmed policy effects](./confirmed-policy-effects.mdx) — **Prototype calculation.** Safety of supported workflow attempts; governed skill trials are a separate evaluation population, never pooled with live work.
|
||||
9. [Cost per accepted outcome](./cost-per-accepted-outcome.mdx) — **Prototype calculation.** Cost of the matured workflow cohort, including unsuccessful attempts and recorded review/rework effort. Missing cost buckets remain partial.
|
||||
10. [Verification confidence and cognitive load](./verification-confidence-and-cognitive-load.mdx) — **Prototype calculation.** Anonymous cohort experience: confidence, workload and usefulness separately, not measured time savings.
|
||||
11. [Workflow-specific business acceptance](./workflow-specific-business-acceptance.mdx) — **Unimplemented metric.** Acceptance against an agreed departmental rubric; requires a defined workflow and independent evidence.
|
||||
|
||||
Human effort is recorded as review/rework inputs where the cost contract supports it; the experience survey describes perception. Neither elapsed delivery time nor token counts establish effort saved. If the current workflow lacks the event semantics required by a contract (for example, governed attempts), record it as not applicable or unavailable with a reason; do not relabel unrelated records to manufacture a baseline.
|
||||
|
||||
### Skills effectiveness & health
|
||||
|
||||
**Are the skills fit for use and kept fit?** Qualify named skills and versions, observe supported use during adoption, and review or retire them in maintenance. Without applicable skills, bundles or enrolled use, skill-specific measures may be **N/A, not zero**. Once a population exists, missing observations are unknown, not evidence of non-use or good health.
|
||||
|
||||
1. [Skill selection precision](./skill-selection-precision.mdx) — **Prototype calculation.** Did the evaluated selection choose an applicable skill? Requires labeled qualification cases.
|
||||
2. [Skill selection recall](./skill-selection-recall.mdx) — **Prototype calculation.** Were expected skills selected? Requires the same labeled qualification population.
|
||||
3. [Supported bundle coverage](./supported-bundle-coverage.mdx) — **Unimplemented metric.** Are expected active installations on supported, non-revoked bundles/adapters?
|
||||
4. [Approved-workflow adoption](./approved-workflow-adoption.mdx) — **Unimplemented metric.** Is the enrolled cohort using approved skills in eligible workflows? Installation/loading is not use.
|
||||
5. [Skills within substantive review window](./skills-within-substantive-review-window.mdx) — **Unimplemented metric.** Do active approved revisions have current owners and substantive review evidence?
|
||||
|
||||
**Task-success population matters.** The canonical validated-task-success contract above measures workflow outcomes, not an evaluation benchmark. A named skill/version can be examined as a predeclared live-workflow slice under that same contract, without counting retries or multiple skills as extra tasks or attributing causality. Sandbox/evaluation cases remain excluded: a named skill/version evaluation must be reported separately with its own declared cases, acceptance evidence and limits. It is not an additional implemented calculation or a second conflicting task-success formula. The canonical page explains this boundary. Likewise, keep policy-effect evaluation trials separate from live attempts.
|
||||
|
||||
Use both tracks together to decide whether to improve, retain or retire a skill. Higher selection scores, wider adoption or current reviews cannot substitute for accepted work, safety, cost and human experience.
|
||||
|
||||
Read [Collection and safeguards](./collection-and-safeguards.mdx) for shared interpretation, event and privacy rules. [Run the synthetic measurement example](./synthetic-example.mdx) for reproducible arithmetic and limits.
|
||||
|
||||
|
||||
@@ -29,6 +29,14 @@ Confirmed independently accepted tasks as a share of all eligible initiated task
|
||||
| Gaming and limits | Split easy tasks or omit starts; reconcile starts to approved backlog and audit acceptance rubrics. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Maturation horizon is workflow-specific and must be approved; no automatic abandonment on timeout. |
|
||||
{/* contract:end */}
|
||||
|
||||
## Workflow outcomes versus skill/version evaluation
|
||||
|
||||
This contract belongs to **Workflow effectiveness**. Use the same matured-start population and independent acceptance rules before and after introducing skills; skills are not required for a workflow outcome to exist. Do not narrow the later denominator to successful skill invocations.
|
||||
|
||||
For **Skills effectiveness & health**, a predeclared slice of live workflow tasks using a named skill/version may reuse this contract unchanged. Keep the parent cohort visible, report exact versions and selection limits, and count each task once across retries, skills and clients. Overlapping skill slices must not be summed. This is descriptive evidence, not proof that the skill caused success.
|
||||
|
||||
A sandbox or qualification evaluation of a named skill/version is a different population and remains excluded by this contract. Declare its cases, skill/version, client/model configuration, acceptance rubric and observed results separately; never combine those cases with live workflow starts or present their results as this prototype's workflow rate. This page does not introduce an implemented skill-evaluation metric or an alternative formula. No applicable skill means that skill-specific evaluation is N/A, not a zero workflow success rate.
|
||||
|
||||
Read the shared [collection and safeguards](./collection-and-safeguards.mdx) before collecting data; see the [synthetic example and limits](./synthetic-example.mdx).
|
||||
|
||||
Return to [Skills Measurement](./index.mdx).
|
||||
|
||||
@@ -13,7 +13,7 @@ import ArchifyDiagram from '@site/src/components/ArchifyDiagram';
|
||||
|
||||
## When should a pilot become a team workflow?
|
||||
|
||||
Expand only when a bounded canary produces independently accepted work and useful feedback. Teach people to verify artifacts and recover without AI, not merely invoke a skill.
|
||||
Expand only when a bounded canary produces independently accepted work and useful feedback. Teach people to verify artifacts and recover without AI, not merely invoke a skill. Compare **Workflow effectiveness** with the current-workflow baseline; observe **Skills effectiveness & health** separately as skills enter use. More adoption does not prove improved work, and before/after trends alone are not causal evidence.
|
||||
|
||||
<ArchifyDiagram name="strategy-adoption" title="Choose a workflow, coach a canary, verify outcomes and improve before expansion" story />
|
||||
|
||||
@@ -60,6 +60,8 @@ Recorded compute spend is **EUR 60 / 1 = EUR 60** per accepted outcome—a **par
|
||||
|
||||
### Revisit selection as adoption expands
|
||||
|
||||
Selection precision/recall qualify named skill versions against labeled cases; they do not measure real-workflow acceptance. Keep any skill/version evaluation separate from the live task-success cohort. Skill-specific measures with no applicable population are N/A, not zero; missing evidence for an existing population remains unknown.
|
||||
|
||||
Keep relevant outcome evidence comparable; add reach measures only if they answer the expanded workflow's questions. For [supported bundle coverage](./measurement/supported-bundle-coverage.mdx), freeze the expected active installation inventory and compare observed bundle/adapter revisions to support and revocation records. Offline or stale entries stay in the denominator. Platform & IT is the proposed weekly and at-revocation reviewer.
|
||||
|
||||
For [approved-workflow adoption](./measurement/approved-workflow-adoption.mdx), freeze the lawfully enrolled participant cohort and distinguish qualified-adapter observations from labeled declarations. Report the observed lower bound with source coverage: missing use is unknown; installation and loading are not use. Data & AI is the proposed monthly reviewer. No invocation quotas or employee rankings. Return to Baseline for a new workflow rather than treating rollout as company-wide metric collection.
|
||||
|
||||
@@ -13,7 +13,7 @@ import ArchifyDiagram from '@site/src/components/ArchifyDiagram';
|
||||
|
||||
## What happens when a skill drifts or causes harm?
|
||||
|
||||
Monitor usefulness and safety throughout a revision's life. Unsafe behavior triggers containment; a correction returns through independent tests and approval, never automatic reinstatement.
|
||||
Monitor both **Workflow effectiveness** (comparable outcomes, safety, cost and human experience) and **Skills effectiveness & health** (selection, supported use and substantive review) throughout a revision's life. Use the evidence to improve, retain or retire skills; adoption or current review dates alone do not prove better work. Unsafe behavior triggers containment; a correction returns through independent tests and approval, never automatic reinstatement.
|
||||
|
||||
<ArchifyDiagram name="strategy-maintenance" title="Monitor, triage and test corrections; promote safely or contain and roll back" story />
|
||||
|
||||
@@ -49,7 +49,7 @@ Owner ratification is required before adoption. Review earlier after incidents o
|
||||
|
||||
## Keep selected evidence useful
|
||||
|
||||
Continue the measures agreed for the workflow in [Baseline](./baseline.mdx) and applied during [Adoption](./skills-adoption.mdx). Revisit relevance when scope changes; the [registry](./measurement/index.mdx) is a source of choices, not a requirement to collect every metric.
|
||||
Continue the measures agreed for the workflow in [Baseline](./baseline.mdx) and applied during [Adoption](./skills-adoption.mdx). Revisit relevance when scope changes; the [registry](./measurement/index.mdx) is a source of choices, not a requirement to collect every metric. Preserve current-workflow comparisons separately from named skill/version evaluations. No applicable skill population is N/A, not zero; missing evidence for existing revisions is unknown and must not improve reported health.
|
||||
|
||||
If substantive-review coverage is selected, join each active approved catalog revision to its owner, substantive review evidence and due date. Apply [review coverage](./measurement/skills-within-substantive-review-window.mdx): revisions with all three current / all active approved revisions × 100. The proposed Data & AI review is monthly and at each release, subject to agreement. Report overdue/ownerless revisions and deprecated or revoked versions still in use—not just a percentage.
|
||||
|
||||
|
||||
@@ -25,12 +25,12 @@ Skills topology connects what skills exist, how they relate, when they are used
|
||||
|
||||
## Put the foundation into practice
|
||||
|
||||
- [Measurement](./measurement/index.mdx) — browse available candidate metrics; inclusion does not require collection or baselining.
|
||||
- [Baseline](./baseline.mdx) — understand a workflow, select relevant metrics and record feasible starting evidence or explicit gaps.
|
||||
- [Measurement](./measurement/index.mdx) — distinguish Workflow effectiveness from Skills effectiveness & health; candidate inclusion does not require collection or baselining.
|
||||
- [Baseline](./baseline.mdx) — understand current workflow outcomes first, select relevant metrics and record feasible starting evidence or explicit gaps; absent skills are not zero performance.
|
||||
- [Governance](./skills-governance.mdx) — define ownership, authority and approval of skills.
|
||||
- [Quality](./skills-quality.mdx) — qualify skills and verify their outcomes before wider use.
|
||||
- [Adoption](./skills-adoption.mdx) — coach bounded workflows with a supported client and a manual path.
|
||||
- [Maintenance](./skills-maintenance.mdx) — review feedback, control versions and recover from unsafe updates.
|
||||
- [Adoption](./skills-adoption.mdx) — compare workflow outcomes and observe skill use separately, with a supported client and a manual path.
|
||||
- [Maintenance](./skills-maintenance.mdx) — monitor both tracks to improve or retire skills, control versions and recover from unsafe updates.
|
||||
|
||||
Proven skills, measurements and working practices support [Stage - 02 - Autonomous Agent](./stage-02-autonomous-agent.mdx); adoption alone does not grant autonomous execution authority.
|
||||
|
||||
|
||||
Reference in new issue
Block a user