eldorado-v1/documentation
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: publish target AI adoption strategy and synthetic measurement examples

Browse Source
This commit is contained in:
jarvis-at-skic committed 2026-10-04 11:08:27 +00:00
1 parent 14ba1aa1ef
commit f1743156b6
22 files changed
+3314 -13
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/strategy/direction.md
-11
View File
@@ -1,11 +0,0 @@
---
title: Direction
description: Define the strategy and direction for an AI-powered Eldorado.
slug: /
---
# Strategy and direction
**Eldorado Powered By AI** is the space for shaping how AI can support Eldorado.
Specific initiatives, priorities, measures of success, and a delivery roadmap remain to be agreed. This section is ready for that work; no proposals here should be treated as approved commitments until explicitly agreed.
docs/strategy/index.mdx
+33
View File
@@ -0,0 +1,33 @@
---
id: index
title: AI adoption strategy
description: Govern skills first; outcomes before activity; explicit target status.
slug: /
---
<div className="strategy-content">
**TO-BE · Target operating model.** Introduction depends on the [roadmap gates](./roadmap.mdx); these pages do not describe installed controls or measured company results.
### Govern skills first, expand authority only with evidence
Eldorado's target is to make AI-assisted work dependable across the organization: useful guidance, clear human ownership, controlled execution and measurable outcomes. The first investment is a governed skills system, not an autonomous replacement for the organization. A skill packages reusable working knowledge, prerequisites and verification steps. It helps an agent perform a task; it does not grant permission or guarantee that the work is correct.
The target combines one logical SDLC coordinating agent with developer choice on local laptops. Copilot, Claude Code and Cline remain client options within a qualified support matrix. They receive the same reviewed portable skill content through client-specific adapters. Repository, CI and release services remain the enforcement points, independently of what any agent says or which local tool produced a change.
The strategy describes the intended operating model and the evidence required to introduce it. It does not establish that these controls are installed, that every client is supported, or that AI has improved Eldorado's performance. The current reference is the documented six-department organization. Rollout begins by validating actual clients, responsibilities, data boundaries and available evidence.
Success means accepted outcomes with maintained quality, controlled risk and understood total cost. Faster activity without reliable outcomes is not success. More skill invocations, generated lines or tool subscriptions are not productivity targets. People remain accountable for product, customer, financial and personnel decisions.
## Navigate the strategy
- [Target operating model](./target-operating-model.mdx) — One logical coordinator; local client choice; independent authority; all departments.
- [Skills governance](./skills-governance.mdx) — Contract, lifecycle, decision rights, adapters, locking, exceptions.
- [Skills quality](./skills-quality.mdx) — Risk tiers, independent evaluation, repeated trials, adversarial conformance.
- [Skills adoption](./skills-adoption.mdx) — Workflow-led coaching, client choice, feedback, manual path.
- [Measurement](./measurement.mdx) — Full KPI contracts, minimal events, illustrative calculations, privacy and limits.
- [Skills maintenance](./skills-maintenance.mdx) — Review, drift, deprecation, rollback and incident handling.
- [Roadmap](./roadmap.mdx) — Evidence-gated phases, no promised dates or budgets.
</div>
docs/strategy/measurement.mdx
+352
View File
@@ -0,0 +1,352 @@
---
id: measurement
title: Measurement
description: Full KPI contracts, minimal events, illustrative calculations, privacy and limits.
---
<div className="strategy-content">
**TO-BE · Target operating model.** Introduction depends on the [roadmap gates](./roadmap.mdx); these pages do not describe installed controls or measured company results.
### Decide what counts before counting
Use the metric contracts below for exact definitions. Every report states its eligible population, observation window, numerator and denominator or distribution sample count, source coverage, exclusions and calculation version. Zero denominators return “not available.” Missing telemetry is unknown, not inactivity, compliance or success. Examples are explicitly synthetic and illustrative.
Begin with task acceptance, repository, CI, deployment and incident systems because they can independently establish outcomes. Add catalog/version inventory and task-level declarations. Client telemetry is supplementary and version-dependent; enabling rich tool details may collect sensitive commands and identifiers. Do not enable it merely to count skills. A normalized adapter may report approved skill identifiers without forwarding raw arguments.
Validated success uses all eligible initiated tasks whose predeclared observation window has matured. Failed and abandoned tasks remain nonsuccess. Unresolved or unobserved outcomes remain in the denominator, with known-outcome coverage shown alongside the conservative confirmed-success share. They are not relabeled as observed failure. A task cancelled before start is excluded; cancellation after start remains visible. Count a task once across retries and multiple skills. For timeliness, also show unfinished work and age; accepted-only lead-time distributions have survivorship limitations.
Track all five service-level DORA measures: commit-to-production lead time, deployment frequency, change fail rate, failed deployment recovery time and deployment rework rate. Do not substitute PR-open-to-merge time or all-incident MTTR. Pair them with matured escaped-quality rates, policy effects, total cost per accepted outcome, support coverage, adoption, human experience and catalog maintenance. Keep customer/business outcomes linked to their departmental hypotheses; do not infer conversion, trust or revenue improvement from delivery telemetry.
### Baseline and interpretation
Before a pilot, freeze eligibility and acceptance rules and verify event joins. Collect prospective no-skill/current-practice evidence over a complete comparable workflow cycle and until the preregistered precision/sample requirement is met. Do not select a convenient end date after seeing results. Retain task class, service, skill, client/model configuration and concurrent changes. A matched or randomized comparison may be used when practical and ethical; label observational trends as descriptive and report confounding, selection and uncertainty.
Elapsed time and human effort are different quantities. Parallel agents may reduce waiting while increasing review or compute. Include failed attempts and human review/rework in cost, do not multiply a multi-skill task's value across every invocation, and do not convert survey perceptions into proven time savings. Small or incomplete populations yield evidence limits, not an ROI percentage.
### Event and privacy contract
Use a versioned event envelope and required fields per event. Source producer plus event ID deduplicates transport retries. Conflicting payloads under the same key are quarantined. Attempts have distinct IDs under one task. Domain IDs deduplicate deployments, defects and approvals even when multiple feeds describe them. Late arrivals produce versioned aggregate corrections; never silently overwrite a published result. Preserve occurrence and receipt times, reject impossible orderings and flag clock skew. Missing terminal events remain unresolved until the work owner reconciles them.
Many-to-many task/attempt/skill/client associations preserve exact immutable versions. Unknown version stays null with a reason; never assign the latest revision retroactively. Client declarations, server facts and self-reports remain separately labeled. A skill-loaded event is evidence of loading only. The minimal prototype can operate without actor identity except where a lawful, necessary adoption denominator requires pseudonymous cohort membership.
The privacy starting proposal is 30 days for raw structured analytics events and 12 months for aggregated trends, subject to the responsible owners' review. Keep the approved catalog/approval record for the supported revision's lifetime and its required audit period, which must be decided separately. Incident evidence uses a restricted, explicitly authorized hold and deletion schedule, not an indefinite analytics exception. Suppress reports below five people and prohibit drill-down or overlapping cohort slices that reconstruct identities. Five is a proposed minimum, not a guarantee against re-identification. No raw prompts, code, screens, customer cases or secrets are collected by default. Publish the purpose, lawful collection basis, access/deletion route and retention before collecting. Consent is not presumed to be an appropriate employment-law basis.
## Metric contracts
Each contract is a prospective collection design. No live source or observed baseline is established. N and D refer to the numerator and denominator shown for that metric.
### Validated task success
Confirmed independently accepted tasks as a share of all eligible initiated tasks in a matured start cohort; a conservative confirmed-success share, not observed failure rate.
| Contract | Definition |
| --- | --- |
| Numerator | Distinct matured task_ids with valid independent task.accepted evidence by as_of |
| Denominator | All distinct eligible task_ids started in window with started_at + preregistered task_observation_hours &lt;= as_of |
| Expression | N / D * 100 |
| Aggregation | percent |
| Unit | % |
| Eligible population | Tasks in a preregistered workflow cohort, started in window; each counted once across retries, skills and clients. |
| Exclusions | Cancelled before start, sandbox/evaluation tasks and out-of-scope workflows; never post-start failures/abandonments. |
| Events and source | task.started; task.accepted; task.failed; task.abandoned; task.cancelled; work-management acceptance record |
| Owner / cadence | Quality Engineering / Weekly pilot; monthly departmental outcome review |
| Missing evidence | Unresolved/missing terminal outcomes remain in denominator with unknown count and known-outcome coverage; do not label them failed. Invalid/missing start excluded from calculation but flagged as orphan evidence. |
| Gaming and limits | Split easy tasks or omit starts; reconcile starts to approved backlog and audit acceptance rubrics. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Maturation horizon is workflow-specific and must be approved; no automatic abandonment on timeout. |
### Skill selection precision
Correct activations among activations on an independently labeled evaluation corpus.
| Contract | Definition |
| --- | --- |
| Numerator | True-positive activations for the requested workflow |
| Denominator | True-positive plus false-positive activations |
| Expression | N / D * 100 |
| Aggregation | percent |
| Unit | % |
| Eligible population | All candidate/client labeled evaluation task-trials with known activation and relevance. |
| Exclusions | Unlabeled/unknown activations excluded with coverage; no cherry-picked failed trials. |
| Events and source | eval.selection; versioned labeled corpus and runner |
| Owner / cadence | Data & AI / Each skill/client qualification |
| Missing evidence | Unknown activation/relevance is unscorable; publish missing count and deny qualification if planned coverage incomplete. |
| Gaming and limits | Choose easy labels or hide negative cases; freeze corpus and independently review labels. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Does not show execution success; precision depends on corpus prevalence. |
### Skill selection recall
Relevant task-trials in which the intended skill activates.
| Contract | Definition |
| --- | --- |
| Numerator | True-positive activations |
| Denominator | True-positive plus false-negative task-trials |
| Expression | N / D * 100 |
| Aggregation | percent |
| Unit | % |
| Eligible population | Independently labeled relevant evaluation task-trials for a skill/client tuple. |
| Exclusions | Irrelevant tasks; missing relevance or activation reported as unscorable, not negative. |
| Events and source | eval.selection; versioned labeled corpus and runner |
| Owner / cadence | Data & AI / Each skill/client qualification |
| Missing evidence | Report evaluation coverage; null if no relevant labeled tasks. |
| Gaming and limits | Overtrigger to maximize recall; always pair with precision and negative cases. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Loading the right skill is not adherence. |
### Supported bundle coverage
Inventoried active installations with verified effective approved, supported and nonrevoked bundle/adapter tuple.
| Contract | Definition |
| --- | --- |
| Numerator | Eligible installation_ids with valid inventory evidence within approved freshness window and matching support matrix |
| Denominator | All expected active installation_ids in the managed pilot inventory |
| Expression | N / D * 100 |
| Aggregation | percent |
| Unit | % |
| Eligible population | Active installation inventory frozen for report, including invisible/offline entries until authoritative offboarding. |
| Exclusions | Verified decommissioned installations and nonpilot assets; lack of telemetry alone never excludes. |
| Events and source | inventory.observed; catalog.published; catalog.revoked; device inventory |
| Owner / cadence | Platform & IT / Weekly and at revocation |
| Missing evidence | Unknown/stale versions count outside numerator, inside denominator. Show inventory discovery coverage versus eligible device roster; if roster unknown metric is provisional. |
| Gaming and limits | Remove silent devices or trust declared versions; reconcile managed inventory and separately label attestation trust. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Even verified file digests do not prove a private model followed the content. |
### Approved-workflow adoption
Eligible participants with at least one approved-skill workflow use; usefulness assessed separately through accepted outcomes and feedback.
| Contract | Definition |
| --- | --- |
| Numerator | Distinct eligible pseudonymous participants with workflow-linked skill.used on approved revision |
| Denominator | Distinct eligible participants in frozen invited cohort |
| Expression | N / D * 100 |
| Aggregation | percent |
| Unit | % |
| Eligible population | Consented or otherwise lawfully enrolled pilot cohort for the period; supported workflow available. |
| Exclusions | Outside pilot and formally recorded leave before period; nonresponders remain eligible, not excluded. |
| Events and source | skill.used; cohort.snapshot; optional qualified client adapter or declaration |
| Owner / cadence | Data & AI / Monthly |
| Missing evidence | Show observed lower bound and telemetry/response coverage; missing usage unknown rather than non-use. |
| Gaming and limits | Invocation quotas and coercion; no target usage count, split server/client/self-report origins. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Self-report is not verified use; installation/loading is not use or benefit. Suppress small groups. |
### Change lead time
Arithmetic mean commit-to-first-successful-production-deployment duration; also publish median/p90 and sample count.
| Contract | Definition |
| --- | --- |
| Numerator | Sum of elapsed hours from committed_at to first successful production deployment for each service+commit |
| Denominator | Count of eligible service+commit pairs with valid mappings |
| Expression | N / D |
| Aggregation | mean |
| Unit | hours/change |
| Eligible population | Changes first successfully deployed to a service in reporting window; rollout retries collapse to same logical deployment. |
| Exclusions | Nonproduction, duplicates, no-op deployment with no new commits; missing mappings shown as coverage loss. |
| Events and source | change.committed; deployment.completed; VCS immutable commit timestamps and release manifest |
| Owner / cadence | Platform & IT / Weekly per service |
| Missing evidence | Missing commit mapping or negative/skewed duration excluded from numeric sample, reported explicitly; no zero substitution. |
| Gaming and limits | Timestamp rewriting and cherry-picking fast work; lock change mapping and show undeployed backlog age. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Not PR cycle time, effort saved, or a complete view of unfinished changes. p90 uses nearest-rank sorted sample. |
### Deployment frequency
Distinct successful logical production deployments per complete service-day of observation.
| Contract | Definition |
| --- | --- |
| Numerator | Distinct service_id+deployment_id successful production deployments in window |
| Denominator | Observed service-days in window |
| Expression | N / D |
| Aggregation | ratio |
| Unit | deployments/service-day |
| Eligible population | Declared production services and complete UTC observation days; report each service before pooled views. |
| Exclusions | Test/staging, failed attempts without production effect, duplicate callbacks and retries of the same rollout. |
| Events and source | deployment.completed; authoritative deployment service |
| Owner / cadence | Platform & IT / Weekly per service |
| Missing evidence | Missing feed periods not zero; report coverage and no unqualified overall rate when period incomplete. |
| Gaming and limits | Split deployments without value; pair with success, rework and failure measures. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. More deployments alone do not establish better performance; affected unsuccessful changes remain in change-fail population. |
### Change fail rate
Production deployment effects requiring unplanned remediation, rollback or hotfix, not solely outage declarations.
| Contract | Definition |
| --- | --- |
| Numerator | Distinct matured production deployment_ids linked to qualifying immediate intervention |
| Denominator | All distinct production deployment_ids that reached production in the matured deployment cohort |
| Expression | N / D * 100 |
| Aggregation | percent |
| Unit | % |
| Eligible population | Deployments with production effect during window and approved intervention follow-up horizon elapsed. |
| Exclusions | Failed preproduction attempts, duplicate callbacks and no-effect retries; failures after production effect stay included. |
| Events and source | deployment.completed; deployment.failed; incident.linked; release and incident records |
| Owner / cadence | Quality Engineering / Weekly; finalized after horizon |
| Missing evidence | Known failures / all eligible deployments is a lower bound where incident linkage incomplete; label provisional and report unknown links. |
| Gaming and limits | Avoid incident links or relabel intervention planned; reconcile rollbacks/hotfixes independently. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Define immediate intervention and follow-up horizon before pilot; no universal incident-to-AI attribution. |
### Failed deployment recovery time
Mean elapsed time from production deployment failure onset to service restoration, with median/p90 and unresolved count/age.
| Contract | Definition |
| --- | --- |
| Numerator | Sum of recovery_at minus failure_started_at in hours for recovered eligible deployment failures |
| Denominator | Count of recovered eligible deployment failures with valid timestamps |
| Expression | N / D |
| Aggregation | mean |
| Unit | hours/failure |
| Eligible population | Production deployment failures beginning in window; latest verified restoration if a recovery is rescinded. |
| Exclusions | Nondeployment incidents; invalid timestamp pairs reported missing, not treated as instantaneous recovery. |
| Events and source | deployment.failed; service.restored; incident service |
| Owner / cadence | Platform & IT / Weekly with unresolved failures |
| Missing evidence | Unresolved failures reported separately and excluded only from recovered-duration distribution; prevent declaring improvement without their visibility. |
| Gaming and limits | Premature recovery declarations or omit hard cases; require operational readback and carry unresolved ages. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Not all-incident MTTR; recovered-only sample has survivorship bias. p90 nearest rank. |
### Deployment rework rate
Unplanned production deployment effects triggered by a production incident as a share of deployment effects.
| Contract | Definition |
| --- | --- |
| Numerator | Distinct incident-linked deployment_ids classified unplanned remediation |
| Denominator | All distinct production deployment_ids with production effect in window |
| Expression | N / D * 100 |
| Aggregation | percent |
| Unit | % |
| Eligible population | Production deployment effects in window with immutable planned/unplanned classification and incident linkage. |
| Exclusions | Nonproduction, no-effect retries and duplicates; do not exclude unclassified changes silently. |
| Events and source | deployment.completed; incident.linked; release intent classification |
| Owner / cadence | Platform & IT / Weekly |
| Missing evidence | Unknown classification stays denominator; observed numerator is lower bound until reconciliation; show classification coverage. |
| Gaming and limits | Call hotfixes planned; freeze reason and independently reconcile incident/deployment links. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Distinct from failure rate: one failed deployment can lead to multiple separate rework deployments. |
### Changes with escaped quality issues
Accepted changes with at least one confirmed post-acceptance defect during approved follow-up, split by severity.
| Contract | Definition |
| --- | --- |
| Numerator | Distinct eligible change_ids with one or more linked confirmed defects in follow-up |
| Denominator | Accepted change_ids with completed predeclared follow-up horizon |
| Expression | N / D * 100 |
| Aggregation | percent |
| Unit | % |
| Eligible population | Accepted changes in acceptance window whose follow-up is complete by as_of. |
| Exclusions | Immature changes shown separately; duplicates/unconfirmed reports excluded but counted as pending triage. |
| Events and source | change.accepted; defect.confirmed; defect tracking and acceptance evidence |
| Owner / cadence | Quality Engineering / Monthly mature cohorts |
| Missing evidence | Unknown linkage is a coverage warning; no fabricated clean outcome. Show absolute defect and severe incident counts as well. |
| Gaming and limits | Shorten follow-up or downgrade severity; freeze horizon and rubric and independently sample defects. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Counts affected changes, not raw defects per change; no presumption defects are AI-caused. |
### Confirmed policy effects
Eligible evaluated attempts with an actual confirmed prohibited effect; report attempts, blocked attempts and effects separately.
| Contract | Definition |
| --- | --- |
| Numerator | Distinct attempt_ids with confirmed prohibited action effect |
| Denominator | All evaluated eligible attempt_ids including failed and abandoned runs |
| Expression | N / D * 100 |
| Aggregation | percent |
| Unit | % |
| Eligible population | Governed skill trials or supported workflow attempts, reported separately for eval and live populations. |
| Exclusions | Out-of-scope runs; detection gaps remain coverage issues, not exclusions from known initiated population. |
| Events and source | attempt.started; policy.violation; gate.blocked; security audit and incident triage |
| Owner / cadence | Architecture & Security / Immediate incident action; weekly aggregates |
| Missing evidence | Unknown outcomes remain visible; zero detected effects never proves safety. Missing audits make rate provisional. |
| Gaming and limits | Hide incidents or inflate harmless trials; absolute severity counts and fixed task mix accompany rate. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Blocked attempts are not actual effects; any critical effect blocks promotion regardless of aggregate rate. |
### Cost per accepted outcome
Allocated total workflow cost including unsuccessful attempts, licenses, compute, tooling and human review/rework divided by accepted outcomes.
| Contract | Definition |
| --- | --- |
| Numerator | Sum allocated costs in one approved reporting currency for all tasks/attempts in matured start cohort |
| Denominator | Distinct independently accepted tasks in same cohort |
| Expression | N / D |
| Aggregation | ratio |
| Unit | currency/accepted outcome |
| Eligible population | Matured eligible started-task cohort, with all attempt costs through as_of and allocated shared period costs. |
| Exclusions | Unrelated projects and documented taxes/transfers outside chosen accounting basis; never failed/abandoned attempts. |
| Events and source | cost.recorded; task.accepted; finance invoices, usage ledger, sampled effort with approved rates |
| Owner / cadence | Finance & Accounting Squad / Monthly with late invoice restatement |
| Missing evidence | Missing cost buckets produce partial estimate plus coverage, not a total. Mixed currency requires dated approved conversion; zero accepted outcomes = null with full cost shown. |
| Gaming and limits | Count tokens only or hide labor/failures; reconcile invoice and allocation totals without double counting. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Allocation and sampled labor carry uncertainty; no ROI without independently measured business benefit. |
### Verification confidence and cognitive load
Three separate anonymous 1–5 item distributions: confidence in verifying output, manageable workload, usefulness. Primary item: verification confidence mean; report full distributions and response rate.
| Contract | Definition |
| --- | --- |
| Numerator | Sum valid 1–5 verification-confidence responses, one per participant per survey wave |
| Denominator | Count valid responses to that item |
| Expression | N / D |
| Aggregation | mean |
| Unit | points (1–5) |
| Eligible population | Invited eligible cohort at fixed survey wave; voluntary responses. |
| Exclusions | Invalid/out-of-range answers and superseded duplicate submission; nonresponders stay in response-rate denominator. |
| Events and source | survey.response; anonymous survey with single-use token and aggregate invitation count |
| Owner / cadence | People & Administration Squad / After pilot workflow cycle; periodic follow-up |
| Missing evidence | Report valid response count / invited count and item nonresponse; no imputation. Suppress people cohorts below five. |
| Gaming and limits | Pressure to respond positively; no manager access to individual responses, avoid identifying free text. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Perception and selection bias; not measured time savings. No composite productivity index. |
### Skills within substantive review window
Active approved skill revisions with an accountable owner and completed substantive review not past due.
| Contract | Definition |
| --- | --- |
| Numerator | Distinct active approved skill revisions with current owner, review evidence and review_due_at &gt;= as_of |
| Denominator | All active approved skill revisions in catalog snapshot |
| Expression | N / D * 100 |
| Aggregation | percent |
| Unit | % |
| Eligible population | Approved revisions currently supported for use at as_of. |
| Exclusions | Experimental and retired revisions; deprecated/revoked in-use counts reported independently, not hidden. |
| Events and source | catalog.published; catalog.reviewed; catalog.deprecated; catalog.revoked; version inventory |
| Owner / cadence | Data & AI / Monthly and each release |
| Missing evidence | Missing owner/date/evidence outside numerator; show ownerless, deprecated-in-use and revoked-in-use absolute counts. |
| Gaming and limits | Rubber-stamp review dates; require linked regression/security/usefulness evidence. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. A current review is not proof of effectiveness; consumer inventory may miss shadow copies. |
### Workflow-specific business acceptance
Independently accepted domain outcomes against a predeclared business rubric, separate from technical completion.
| Contract | Definition |
| --- | --- |
| Numerator | Eligible completed-observation task_ids meeting the department-specific outcome rubric |
| Denominator | All eligible initiated workflow task_ids with matured business-outcome horizon |
| Expression | N / D * 100 |
| Aggregation | percent |
| Unit | % |
| Eligible population | One named departmental workflow and predeclared start cohort; never pool unlike rubrics. |
| Exclusions | Cancelled-before-start and unrelated workflows; unsuccessful or abandoned eligible work remains. |
| Events and source | task.started; business.accepted; departmental case/content/reconciliation/onboarding review record |
| Owner / cadence | Product Management / Each pilot cycle with owning department |
| Missing evidence | Unknown follow-up stays denominator with unresolved count; no claim actual business outcome changed without source evidence. |
| Gaming and limits | Weaken rubric or count drafts as customer benefit; lock rubric and independently audit acceptance. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Domain owner signs acceptance: Operations service accuracy; Marketing claims/rights; Finance reconciliation accuracy; HR policy correctness; Product usability/value. Not revenue uplift. |
## Run the synthetic measurement example
**Executed synthetic example — not Eldorado results.** The bundled fixture produces 1 independently accepted task from 4 matured starts: 25% confirmed success. One unresolved task remains in that denominator; known-outcome coverage is 75%. Selection precision and recall are each 1 / 2 = 50%. Allocated synthetic cost is EUR 60 / 1 accepted outcome = EUR 60 per accepted outcome. These deliberately small samples demonstrate arithmetic and missingness, not a baseline, target or ROI claim.
From the private documentation checkout, run `node scripts/ai-kpi.mjs` and `node --test scripts/ai-kpi.test.mjs`. The fixture `scripts/fixtures/ai-kpi-synthetic.json` is labeled synthetic; every result carries the same label and calculation version. This prototype has no network collectors and does not enable employee telemetry.
The executable covers validated success, selection precision/recall, change fail and rework rates, escaped quality, policy effects, cost and human experience. Other catalog metrics remain contracts rather than implemented collectors; output lists the omissions. Snapshot inputs are simplified, pre-reconciled records, not a production ingestion system. Conflicting duplicates reject the calculation rather than silently choosing a value.
</div>
docs/strategy/roadmap.mdx
+54
View File
@@ -0,0 +1,54 @@
---
id: roadmap
title: Roadmap
description: Evidence-gated phases, no promised dates or budgets.
---
<div className="strategy-content">
**TO-BE · Target operating model.** Introduction depends on the [roadmap gates](./roadmap.mdx); these pages do not describe installed controls or measured company results.
### Foundation
**Entry evidence** — Strategy scope accepted for planning; existing teams identified.
**Work** — Inventory actual client hosts/builds, provider/data terms, repositories, authority, owners and event availability; select low-risk workflows and metric definitions.
**Exit evidence and hold condition** — Named accountable people and independent reviewers, bounded pilot charter, forbidden actions, data/retention decision and reproducible baseline plan. Hold if accountability or lawful collection is unresolved.
### Governed skills pilot
**Entry evidence** — Foundation evidence approved; no privileged autonomous effects.
**Work** — Build a small registry, immutable contract, client adapters, evaluations, coaching and synthetic measurement verification.
**Exit evidence and hold condition** — Qualified client/skill tuples, mandatory safety checks, independent task acceptance, reconciled coverage and user feedback. Hold or narrow scope if quality or data coverage is inadequate.
### Reliable distribution
**Entry evidence** — Pilot quality and usefulness support continuing.
**Work** — Canary updates, effective-version readback, version locking, exception expiry, stale/offline handling and rollback.
**Exit evidence and hold condition** — Tested revocation and recovery, no bypass of central privileged gates, owners can support the catalog. Hold if effective versions or controls cannot be verified.
### Logical SDLC coordination
**Entry evidence** — Distribution, authority and measurement evidence stable.
**Work** — Observe and assemble evidence first, then propose and route work within scoped identities and budgets.
**Exit evidence and hold condition** — Exact-artifact approvals, idempotency, duplicate/outage recovery, separation of duties, audit and kill-switch drills pass; humans can take over. No authority expansion based on usage alone.
### Departmental expansion
**Entry evidence** — Each department supplies a workflow owner, outcome, data boundary and capacity.
**Work** — Add Operations, Marketing, Finance & Accounting and HR & Administration workflows with domain-specific review; Product and Technology continue quality improvement.
**Exit evidence and hold condition** — Departmental accepted outcomes, guardrails, total cost and experience justify continuation; sensitive decisions stay human. Stop or roll back workflows without adequate evidence.
There are no invented launch dates, budgets, staffing commitments or guaranteed gains. Phase gates permit narrowing or stopping. Re-evaluate earlier phases when the underlying client, provider, data scope or authority changes.
</div>
docs/strategy/skills-adoption.mdx
+20
View File
@@ -0,0 +1,20 @@
---
id: skills-adoption
title: Skills adoption
description: Workflow-led coaching, client choice, feedback, manual path.
---
<div className="strategy-content">
**TO-BE · Target operating model.** Introduction depends on the [roadmap gates](./roadmap.mdx); these pages do not describe installed controls or measured company results.
Start with a small set of useful workflows chosen by the existing teams. Product Management and the departmental owner identify the outcome; Data & AI helps turn it into a skill; Quality Engineering establishes acceptance; Platform & IT qualifies the client path; Architecture & Security clears the data boundary. People & Administration Squad coordinates learning and accessible feedback.
Onboarding should teach people to select the right skill, inspect the actual artifact, recognize untrusted instructions, use the exception and incident routes and recover without AI. A practice task ends with verified work and an explanation of limits, not just a logged invocation. Demonstrate the same portable skill on the supported local clients without promising identical agent behavior.
Use short worked examples, peer pairing, office hours and a clear catalog search path. Gather reasons for non-use and rejected suggestions without penalizing them. Preserve a manual path. Training feedback improves the skill only through review, not automatic self-modification of the approved catalog. Departmental workflow outcomes matter more than uniform uptake.
No developer rankings, individual productivity scores, minimum invocation quotas or inference of effort from token counts are part of this strategy. Report cohort-level evidence with privacy protection and balanced quality, flow, risk and cost. A participant may reasonably stop using a skill that creates more work or uncertainty.
</div>
docs/strategy/skills-governance.mdx
+73
View File
@@ -0,0 +1,73 @@
---
id: skills-governance
title: Skills governance
description: Contract, lifecycle, decision rights, adapters, locking, exceptions.
---
<div className="strategy-content">
**TO-BE · Target operating model.** Introduction depends on the [roadmap gates](./roadmap.mdx); these pages do not describe installed controls or measured company results.
### A contract that travels, controls that stay authoritative
Maintain one catalog entry per stable skill ID, with immutable revisions and a concise portable `SKILL.md` body. Required content includes when to use and when not to use the skill, prerequisites, bounded steps, expected artifacts, verification, failure handling and escalation. The catalog contract below defines owner and independent reviewer, risk tier, supported clients and versions, dependencies, digest, provenance and license, evaluations, approval, review date, deprecation and rollback.
The release manifest pins the content, dependencies and adapter revisions by full immutable commit or digest. Names use an `eldorado-` prefix to reduce collisions. Friendly versions aid communication but never replace the digest. No arbitrary upstream automatic update is admitted into the approved bundle. Changes to scripts, resources or dependencies count as skill changes, not harmless documentation edits.
Adapters place content in the qualified client's discovery paths and implement only verified client-specific behavior. Qualification records the host, exact build, OS, model configuration, discovery, explicit invocation, collisions, disabled skills, permission behavior, context compaction and reload behavior. A common file format is not common precedence or a portable hooks API. Claude and Cline skill-name precedence differs from rule/memory behavior; Copilot support varies by surface. Test the actual build, not the brand name.
### Lifecycle and decision rights
A proposal names the problem, owner, data scope and testable outcome. Experimental revisions run only in approved disposable environments. Promotion requires independent review, static and executable dependency checks, per-client behavioral evaluations and a bounded canary. Approval records bind the exact artifact and permitted uses. Deprecated revisions carry a migration path and deadline; retired revisions are unavailable for new work. Any active revision can be quarantined immediately.
The author never supplies the only approval. Data & AI maintains catalog state; the owning team is accountable for content; Quality Engineering approves evaluation evidence; Architecture & Security approves higher-risk boundaries; Platform & IT verifies distribution and revocation. A business owner accepts workflow outcomes. These functions can be covered by existing staff, but incompatible author/approver duties must remain separate for the same release.
An exception contains its ID, skill/client revision, scope, business reason, risk owner, compensating checks, approver, start and expiry, affected users/projects and rollback route. It does not disable protected release, payment or identity controls. Expired or unverifiable exceptions deny privileged actions. Repeated exceptions trigger a supported-design change rather than permanent informal bypass.
### Distribution, stale clients and rollback
Stage a verified bundle atomically, check its digest, load it through the qualified adapter and read back its effective revision. Test already-open sessions; changing files is not proof of reloading instructions. Record immutable last-known-good content and adapter dependencies. Canary upgrades precede wider distribution.
A cached verified bundle may support explicitly permitted low-risk work if its model and tool dependencies are available. It does not make a cloud-backed client work offline. Stale or unsupported sessions cannot obtain centrally mediated privileged access. Reconnection revalidates support, revocation and policy before credentials are issued. Central artifact review protects delivery even when a laptop declaration is absent or false; a client report cannot prove which private prompt was actually followed.
Rollback restores the approved manifest and dependencies, invalidates affected sessions, revokes unsafe revisions, runs a smoke test and verifies effective state. It cannot undo data disclosure or commands already executed. Those require incident response.
## Catalog contract
| Contract | Definition |
| --- | --- |
| skill_id | Stable globally unique eldorado-&lt;domain&gt;-&lt;purpose&gt;; never reused for a different purpose |
| name | Human-readable name |
| purpose | Problem, expected outcome and intended users |
| triggers | Positive and negative invocation conditions |
| owner | Existing department/team plus assigned accountable person before approval |
| reviewer | Independent person and existing team; cannot equal release author |
| backup | Named coverage within existing team before production support |
| risk_tier | R1\|R2\|R3; maximum implied by data, tool and effect scope |
| data_classification | Allowed input/output/provider boundary and forbidden data |
| authority_envelope | Allowed resources/operations and forbidden effects; enforced externally |
| supported_clients | Array of client, host, exact tested builds, OS, model configuration, adapter digest and qualification evidence |
| dependencies | All skill/resource/script/tool/MCP/plugin dependencies pinned by immutable digest plus license and source |
| revision | Full immutable source commit and content/artifact digest; friendly semver optional |
| tests_evals | Suite/corpus/rubric/runner/judge revisions, client tuple, seeds or trial IDs, results and independent evidence refs |
| approval | Approver identities, exact digest and policy revision, approved scope, time and expiry/invalidation condition |
| provenance_license | Origin repository/author, upstream revision, acquisition/review history, rights/license for content and bundled resources |
| maintenance_date | Last substantive review timestamp plus next review due and trigger conditions |
| deprecation | State, reason, replacement, migration instructions, consumers and retirement deadline if applicable |
| rollback | Approved known-good bundle+adapter+dependency digests, revocation/reload procedure and tested evidence |
| procedure | Prerequisites, bounded steps, expected artifacts, verification, failure handling, escalation |
## Promotion and retirement
| Transition | Required evidence |
| --- | --- |
| proposed → experimental | Owner, purpose, risk/data scope, disposable environment and preregistered test plan |
| experimental → approved | Independent review, all mandatory tests, qualified clients, canary, exact-digest approval |
| approved → deprecated | Owner gives migration/replacement and retirement conditions |
| deprecated → retired | Consumers reconciled or explicit risk-approved cutover; no new invocation |
| any active state → quarantined | Incident or unsafe/unknown provenance; central deny immediately |
| quarantined → experimental | Reviewed correction; new immutable revision; requalify, never auto-restore approval |
</div>
docs/strategy/skills-maintenance.mdx
+28
View File
@@ -0,0 +1,28 @@
---
id: skills-maintenance
title: Skills maintenance
description: Review, drift, deprecation, rollback and incident handling.
---
<div className="strategy-content">
**TO-BE · Target operating model.** Introduction depends on the [roadmap gates](./roadmap.mdx); these pages do not describe installed controls or measured company results.
Every approved skill has an accountable owning team, independent reviewer, review due date, supported client matrix and rollback bundle. Review both usefulness and safety, not just metadata freshness. Proposed maximum review intervals are recorded by risk tier; incidents, dependency changes, model/client drift or repeated failures trigger earlier review.
Data & AI reviews catalog health with the owning teams. Quality Engineering samples accepted outputs and adds confirmed failures to regression sets without exposing sensitive content. Platform & IT watches unsupported, deprecated and revoked revisions and tests distribution recovery. Architecture & Security reviews emerging misuse and exception expiry. Remove redundant or unused entries deliberately, after examining whether poor discoverability or fit explains non-use.
A reported unsafe behavior opens an incident with artifact, policy and client versions. Architecture & Security coordinates containment; Platform & IT revokes authority, stops scheduling and quarantines affected revisions; the owning department handles customer or business consequences; Quality Engineering reproduces the failure; Data & AI coordinates catalog correction and trend analysis. Preserve minimal restricted evidence and involve People & Administration Squad for employee-data concerns and Finance & Accounting Squad for financial effects. Restart requires a reviewed fix, regression evidence, verified rollback/revocation and accountable human authorization. Closing the incident does not erase the historical failure from measurement.
## Proposed maximum review intervals
Owner ratification is required before adoption. Review earlier after incidents or material dependency changes.
| Tier | Maximum days between substantive reviews |
| --- | --- |
| R1 | 90 |
| R2 | 60 |
| R3 | 30 |
</div>
docs/strategy/skills-quality.mdx
+60
View File
@@ -0,0 +1,60 @@
---
id: skills-quality
title: Skills quality
description: Risk tiers, independent evaluation, repeated trials, adversarial conformance.
---
<div className="strategy-content">
**TO-BE · Target operating model.** Introduction depends on the [roadmap gates](./roadmap.mdx); these pages do not describe installed controls or measured company results.
Quality has four separate questions: was the right skill selected, was the procedure executed safely, did the artifact pass independent checks, and did the workflow produce useful outcomes? A successful invocation answers none of the remaining questions by itself.
The proposed qualification contract uses three tiers. R1 is read-only or drafting with nonsensitive approved material; R2 creates code or material with customer impact but no privileged external effect; R3 touches privileged, financial, sensitive-personal or production decisions. R3 remains human-authorized and out of the initial autonomous scope.
Every tier requires all mandatory deterministic safety, schema, authorization and provenance checks to pass, with no observed critical unauthorized action. An observed critical effect blocks promotion. The proposed pilot evaluation floors below set success thresholds; these are design defaults for owner ratification, not externally validated guarantees or approved service levels. If sample coverage, acceptance criteria or responsible reviewers are missing, qualification remains incomplete. No aggregate score can trade off a failed mandatory safety check.
Use sanitized, predeclared task sets containing positive triggers, near misses, negative cases and the normal workload mix. Compare no-skill, current-approved and candidate paths with the same starting artifacts; repeat stochastic trials and retain failed and abandoned attempts. Independently inspect output state. Deterministic tests grade checkable constraints; calibrated humans grade usefulness, maintainability and experience. A model judge is supplementary, versioned and checked against human disagreements, never the sole safety or promotion authority.
Include adversarial repository text, malicious referenced code, wrong permissions, client toggles, name shadowing, unsupported versions, hook timeout/crash/malformed response, stale sessions, offline recovery and rollback. Run central checks against forged/missing client declarations. Hook presence is not fail-closed enforcement: failure semantics must be tested on each client, and central authorization remains independent.
Report denominators, task mix, trial counts, per-client results and uncertainty. A passing benchmark is evidence within a tested envelope, not proof that all future tasks are safe. Material model, client, adapter or dependency changes trigger requalification.
## Proposed qualification floors
These are local design defaults for responsible-owner ratification, not approved service levels or guarantees. All mandatory checks must pass regardless of aggregate score.
### R1
**Distinct tasks** — 30
**Trials per task/client** — 3
**Artifact success** — 90%
**Selection precision / recall** — 90% / 90%
### R2
**Distinct tasks** — 60
**Trials per task/client** — 3
**Artifact success** — 95%
**Selection precision / recall** — 95% / 90%
### R3
**Distinct tasks** — 100
**Trials per task/client** — 5
**Artifact success** — 99%
**Selection precision / recall** — 99% / 95%
Artifact success denominator is every planned executed trial, retaining failures and abandonment. Trials of same task are correlated: show per-task and per-client strata; do not inflate independent sample size. Selection floors use independent labels. Report 95% uncertainty intervals clustered by task for outcome comparisons; no regression versus current approved path beyond a preregistered owner-approved margin. Missing margin/precision rule prevents promotion.
</div>
docs/strategy/target-operating-model.mdx
+113
View File
@@ -0,0 +1,113 @@
---
id: target-operating-model
title: Target operating model
description: One logical coordinator; local client choice; independent authority; all departments.
---
import ArchifyDiagram from '@site/src/components/ArchifyDiagram';
<div className="strategy-content">
## Coordination without transferred authority
The diagram is an overview; the equivalent delivery path below remains readable on narrow screens. Expand the diagram to inspect its trust boundaries and use its zoom controls for individual relationships.
1. Accountable people authorize the work scope; one logical coordinator routes tasks.
2. The reviewed registry supplies pinned skills and qualified adapters to coordinator and local clients.
3. Laptop contributors propose changes; independent repository and CI checks verify the exact revision.
4. A human approves that artifact; only the release service holds production credentials.
5. Restricted audit and measurement evidence returns to people and informs the next authorized task.
<ArchifyDiagram name="ai-adoption-target" title="TO-BE AI adoption: human accountability, no direct production authority for laptops or coordinator" />
**TO-BE · Target operating model.** Introduction depends on the [roadmap gates](./roadmap.mdx); these pages do not describe installed controls or measured company results.
### One coordination function, distributed contributions
Data & AI owns the coordinating function, with the existing AI Architect function connecting it to Technology leadership. The logical coordinator maintains approved work records, dependencies, evidence requests, bounded retries and escalation. It can summarize, propose work and route decisions. It cannot declare its own work approved, change protected checks, authorize production release or replace the responsible human.
One logical coordinator does not mean one fragile process. A durable task ledger, explicit state transitions, leases and idempotency keys allow recovery or multiple replicas without duplicate delivery. Inference outages stop new agent decisions, not the organization's ability to take over. People can resume work through the normal deterministic workflow. Evidence and authorization services fail closed for privileged actions; missing evidence never becomes permission.
The delivery path is: an approved problem and acceptance criteria; an authorized task; local or isolated worker proposals; independently reviewed repository changes; exact-revision CI evidence; a human release decision; execution by the release service; then operational feedback. Product Management accepts the outcome hypothesis and business result. Product Design reviews experience and accessibility. Product Engineering owns the code and maintainability. Quality Engineering assesses quality independently. Architecture & Security assesses risk. Platform & IT operates the delivery boundary. No arrow from a laptop or coordinator goes directly to production authority.
### Trust and authority boundaries
People authenticate to the work system; the coordinator receives only the project context it needs. Untrusted issue text, repository content, web pages and tool output are task data, never an authorization source. Local clients operate under device controls and project-scoped access. Hard containment requires OS/container isolation, filesystem and network restrictions and separate short-lived identities; a skill folder, prompt or working directory is not a sandbox.
The registry distributes reviewed immutable content and adapters, not ambient credentials. Executable scripts, plugins, MCP servers and dependencies are separately reviewed and pinned. Model-provider routing is allowlisted for the data classification, with approved retention and training-use terms before sensitive data crosses that boundary. Fallback to another provider must not silently change the data policy.
Central controls verify protected-branch review, actual artifact identity, exact commit and policy revision, required checks and a scoped human approval. Candidate changes cannot edit their own authoritative gate or reuse approval for a different artifact. Deployment credentials belong to the release service, not the local agent. Approval expires or is invalidated by changed content. Emergency stop revokes mutation credentials and stops scheduling independently of the model, preserves evidence and enables human takeover.
### Department contributions
### Product / Product Management
**Contribution and retained human decision** — Own pilot priorities, acceptance criteria and outcome hypotheses; decide product value.
**First bounded workflow and evidence** — Draft acceptance criteria from sanitized discovery notes; independent completeness review and accepted-task success.
### Product / Product Design
**Contribution and retained human decision** — Own design quality, accessibility and research handling.
**First bounded workflow and evidence** — Review a proposed flow against an accessibility checklist; verify findings against the interface.
### Technology / Product Engineering
**Contribution and retained human decision** — Own skill authoring, code review and maintainability; remain accountable for generated code.
**First bounded workflow and evidence** — Test scaffolding and change summaries in a disposable branch; independent tests and escaped-quality follow-up.
### Technology / Platform & IT
**Contribution and retained human decision** — Own supported client inventory, distribution, identity, isolation, CI/release integration, transport and emergency revocation.
**First bounded workflow and evidence** — Qualify a pinned bundle and exercise rollback; inventory and exact-revision gate evidence.
### Technology / Architecture & Security
**Contribution and retained human decision** — Own risk classification, data boundaries, security review and bounded exceptions.
**First bounded workflow and evidence** — Review injected-input and unauthorized-tool cases; independent control evidence and policy-effect counts.
### Technology / Quality Engineering
**Contribution and retained human decision** — Own regression/adversarial suites, rubric calibration and acceptance evidence.
**First bounded workflow and evidence** — Evaluate candidate versus approved skills on repeated representative tasks; retain all failures.
### Technology / Data & AI
**Contribution and retained human decision** — Own catalog coordination, coordinator evaluation and measurement design, through existing AI enablement responsibilities.
**First bounded workflow and evidence** — Curate a small skill set and analyze a prospective pilot; no new AI department.
### Operations / Operations Squad
**Contribution and retained human decision** — Own service acceptance, disputes, fraud, refunds and customer communications.
**First bounded workflow and evidence** — Draft an answer from sanitized resolved cases; Operational Quality workstream checks correctness before any send. No automated dispute/refund decision.
### Marketing / Marketing Squad
**Contribution and retained human decision** — Own brand, factual claims, content rights, publication and spend approvals.
**First bounded workflow and evidence** — Draft a campaign brief against approved brand material; human claim/rights review before publication.
### Finance & Accounting / Finance & Accounting Squad
**Contribution and retained human decision** — Own allocation of total costs, financial evidence and segregation of approvals.
**First bounded workflow and evidence** — Explain a sanitized reconciliation exception; no posting, payout or payment authority.
### HR & Administration / People & Administration Squad
**Contribution and retained human decision** — Own coaching, acceptable use, feedback privacy and people decisions; partner with Platform & IT on access lifecycle.
**First bounded workflow and evidence** — Draft onboarding guidance from approved policy; human accuracy review. No hiring or performance decision automation.
Product keeps two teams; Technology keeps five; the other four departments keep one squad each. Coordination, skill ownership, approval and cover are assignments inside these teams, not new standing departments, roles or headcount. Department leaders retain accountability and designate real approvers before a pilot starts.
See the existing [organization structure](/organization/structure/), [department collaboration](/organization/collaboration/) and [SDLC responsibilities](/organization/sdlc/).
</div>