61 lines
3.8 KiB
Plaintext
61 lines
3.8 KiB
Plaintext
---
|
|
id: skills-quality
|
|
title: Skills quality
|
|
description: Risk tiers, independent evaluation, repeated trials, adversarial conformance.
|
|
---
|
|
|
|
|
|
<div className="strategy-content">
|
|
|
|
**TO-BE · Target operating model.** Introduction depends on the [roadmap gates](./roadmap.mdx); these pages do not describe installed controls or measured company results.
|
|
|
|
Quality has four separate questions: was the right skill selected, was the procedure executed safely, did the artifact pass independent checks, and did the workflow produce useful outcomes? A successful invocation answers none of the remaining questions by itself.
|
|
|
|
The proposed qualification contract uses three tiers. R1 is read-only or drafting with nonsensitive approved material; R2 creates code or material with customer impact but no privileged external effect; R3 touches privileged, financial, sensitive-personal or production decisions. R3 remains human-authorized and out of the initial autonomous scope.
|
|
|
|
Every tier requires all mandatory deterministic safety, schema, authorization and provenance checks to pass, with no observed critical unauthorized action. An observed critical effect blocks promotion. The proposed pilot evaluation floors below set success thresholds; these are design defaults for owner ratification, not externally validated guarantees or approved service levels. If sample coverage, acceptance criteria or responsible reviewers are missing, qualification remains incomplete. No aggregate score can trade off a failed mandatory safety check.
|
|
|
|
Use sanitized, predeclared task sets containing positive triggers, near misses, negative cases and the normal workload mix. Compare no-skill, current-approved and candidate paths with the same starting artifacts; repeat stochastic trials and retain failed and abandoned attempts. Independently inspect output state. Deterministic tests grade checkable constraints; calibrated humans grade usefulness, maintainability and experience. A model judge is supplementary, versioned and checked against human disagreements, never the sole safety or promotion authority.
|
|
|
|
Include adversarial repository text, malicious referenced code, wrong permissions, client toggles, name shadowing, unsupported versions, hook timeout/crash/malformed response, stale sessions, offline recovery and rollback. Run central checks against forged/missing client declarations. Hook presence is not fail-closed enforcement: failure semantics must be tested on each client, and central authorization remains independent.
|
|
|
|
Report denominators, task mix, trial counts, per-client results and uncertainty. A passing benchmark is evidence within a tested envelope, not proof that all future tasks are safe. Material model, client, adapter or dependency changes trigger requalification.
|
|
|
|
## Proposed qualification floors
|
|
|
|
These are local design defaults for responsible-owner ratification, not approved service levels or guarantees. All mandatory checks must pass regardless of aggregate score.
|
|
|
|
### R1
|
|
|
|
**Distinct tasks** — 30
|
|
|
|
**Trials per task/client** — 3
|
|
|
|
**Artifact success** — 90%
|
|
|
|
**Selection precision / recall** — 90% / 90%
|
|
|
|
### R2
|
|
|
|
**Distinct tasks** — 60
|
|
|
|
**Trials per task/client** — 3
|
|
|
|
**Artifact success** — 95%
|
|
|
|
**Selection precision / recall** — 95% / 90%
|
|
|
|
### R3
|
|
|
|
**Distinct tasks** — 100
|
|
|
|
**Trials per task/client** — 5
|
|
|
|
**Artifact success** — 99%
|
|
|
|
**Selection precision / recall** — 99% / 95%
|
|
|
|
Artifact success denominator is every planned executed trial, retaining failures and abandonment. Trials of same task are correlated: show per-task and per-client strata; do not inflate independent sample size. Selection floors use independent labels. Report 95% uncertainty intervals clustered by task for outcome comparisons; no regression versus current approved path beyond a preregistered owner-approved margin. Missing margin/precision rule prevents promotion.
|
|
|
|
</div>
|