jarvis-at-skic
f7a68d7e37
Build and publish Eldorado documentation / build (pull_request) Successful in 42s
45 lines
3.7 KiB
Plaintext
45 lines
3.7 KiB
Plaintext
---
|
|
id: validated-task-success
|
|
slug: /measurement/validated-task-success
|
|
title: Validated task success
|
|
description: Definition, collection evidence and limitations for validated task success.
|
|
---
|
|
|
|
<div className="strategy-content">
|
|
|
|
**Target operating model — recommended contracts, not installed collection.** All owner and cadence entries below are proposals to confirm before the pilot, not assigned responsibilities or approved schedules.
|
|
|
|
**Prototype calculation — no live telemetry.** The synthetic example calculates this metric; it has no network collectors.
|
|
|
|
{/* contract:start */}
|
|
Confirmed independently accepted tasks as a share of all eligible initiated tasks in a matured start cohort; a conservative confirmed-success share, not observed failure rate.
|
|
|
|
| Contract | Definition |
|
|
| --- | --- |
|
|
| Numerator | Distinct matured task_ids with valid independent task.accepted evidence by as_of |
|
|
| Denominator | All distinct eligible task_ids started in window with started_at + preregistered task_observation_hours <= as_of |
|
|
| Expression | N / D * 100 |
|
|
| Aggregation | percent |
|
|
| Unit | % |
|
|
| Eligible population | Tasks in a preregistered workflow cohort, started in window; each counted once across retries, skills and clients. |
|
|
| Exclusions | Cancelled before start, sandbox/evaluation tasks and out-of-scope workflows; never post-start failures/abandonments. |
|
|
| Events and source | task.started; task.accepted; task.failed; task.abandoned; task.cancelled; work-management acceptance record |
|
|
| Owner / cadence | Quality Engineering / Weekly pilot; monthly departmental outcome review |
|
|
| Missing evidence | Unresolved/missing terminal outcomes remain in denominator with unknown count and known-outcome coverage; do not label them failed. Invalid/missing start excluded from calculation but flagged as orphan evidence. |
|
|
| Gaming and limits | Split easy tasks or omit starts; reconcile starts to approved backlog and audit acceptance rubrics. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Maturation horizon is workflow-specific and must be approved; no automatic abandonment on timeout. |
|
|
{/* contract:end */}
|
|
|
|
## Workflow outcomes versus skill/version evaluation
|
|
|
|
This contract belongs to **Workflow effectiveness**. Use the same matured-start population and independent acceptance rules before and after introducing skills; skills are not required for a workflow outcome to exist. Do not narrow the later denominator to successful skill invocations.
|
|
|
|
For **Skills effectiveness & health**, a predeclared slice of live workflow tasks using a named skill/version may reuse this contract unchanged. Keep the parent cohort visible, report exact versions and selection limits, and count each task once across retries, skills and clients. Overlapping skill slices must not be summed. This is descriptive evidence, not proof that the skill caused success.
|
|
|
|
A sandbox or qualification evaluation of a named skill/version is a different population and remains excluded by this contract. [Quality](../skills-quality.mdx) defines artifact-success and revision-regression evidence: declare cases, skill/version, client/model configuration, acceptance rubric and observed results separately; never combine those cases with live workflow starts or present their results as this prototype's workflow rate. This page does not introduce an implemented skill-evaluation metric or an alternative formula. No applicable skill means that skill-specific evaluation is N/A, not a zero workflow success rate.
|
|
|
|
Read the shared [collection and safeguards](./collection-and-safeguards.mdx) before collecting data; see the [synthetic example and limits](./synthetic-example.mdx).
|
|
|
|
Return to [Skills Measurement](./index.mdx).
|
|
|
|
</div>
|