docs: publish target AI adoption strategy and synthetic measurement examples

This commit is contained in:
jarvis-at-skic committed 2026-10-04 11:08:27 +00:00
1 parent 14ba1aa1ef
commit f1743156b6
22 files changed
+3314 -13

No files matched your search

+229
View File
@@ -0,0 +1,229 @@
# Eldorado AI adoption — research handoff
Research only. Retrieved 4 October 2026 UTC. This document separates documented vendor capability, empirical evidence, and recommendations. Nothing here certifies an Eldorado rollout, installed client, enabled telemetry, staffing appointment, measured baseline, or realized productivity improvement. Sources and literal supporting passages are in `sources.json`; fetched text, retrieval outcomes and citation ledger are retained alongside it.
## 1. Local organizational baseline and scope
Read-only remote fetch and status inspection found a clean documentation checkout on `main`, with HEAD and `origin/main` both `14ba1aa1ef66c91525135f743d15c77e32db78d3`. Inspected all three current Organization pages: `structure.mdx`, `collaboration.mdx`, `sdlc.mdx`, plus channel wiki `/opt/data/wiki/channels/1555510466904727622.md`. This is documentation evidence, not independent verification of actual personnel, systems, or deployed controls. No repository file was changed.
Preserve the existing structure:
- Product: Product Management and Product Design.
- Technology: Product Engineering, Platform & IT, Architecture & Security, Quality Engineering, Data & AI.
- Operations: Operations Squad; Marketing: Marketing Squad; Finance & Accounting: Finance & Accounting Squad; HR & Administration: People & Administration Squad.
- Existing leadership documentation locates AI enablement coordination with the AI Architect within Data & AI, alongside established engineering leadership. Do not invent a new AI department, standing committee, delivery team, or confirmed headcount.
The requested target is ONE logical SDLC coordinating agent, with people using local Copilot, Claude Code or Cline agents. Logical coordination does not require a single fragile process and does not transfer human accountability to a model. The recommendations below develop this target; they are not a description of present operations.
## 2. Principal conclusions
1. Standardize a portable skill content contract, not the fiction of one universal client policy. Agent Skills defines `SKILL.md`, metadata and resources; `allowed-tools` is experimental, and the discovery directory is not mandated by the format.[1][21]
2. Treat instructions as guidance. GitHub explicitly warns of nondeterministic instruction following, and Claude describes memory files as context rather than enforced configuration.[3][5] Recommended enforcement belongs in independently administered repository, CI, artifact and release controls.
3. Maintain client-specific adapters and an explicit supported version matrix. Current official docs disagree even about collision precedence: Claude enterprise > personal > project; Cline global skill > project skill; Cline rules use the opposite workspace-over-global rule.[4][8][9]
4. Assess skill selection, execution, artifact quality and business outcomes separately. Repeated trials and environment-state verification matter more than an agent reporting success.[11]
5. Measure a balanced portfolio of flow, quality, risk, cost and human experience. DORA now uses five delivery metrics; SPACE explicitly rejects a single productivity metric.[12][24] Adoption counts or accepted code are not causal productivity evidence.
6. Use comparative pilots, not an imported percentage ROI. Controlled studies have reported substantial speedups and slowdowns in different settings, and METR's newer study explains why selection and concurrent-agent timing make current effects difficult to estimate.[16][17][22]
## 3. Compatibility and enforcement matrix
Every capability below is documented upstream, not exercised in Eldorado. A supported client means a particular host, version, model configuration and OS combination that passes local conformance tests; a product brand alone is insufficient.
| Dimension | GitHub Copilot | Claude Code | Cline |
| --- | --- | --- | --- |
| Skills and scope | Project `.github/skills`, `.claude/skills`, `.agents/skills`; personal `~/.copilot/skills`, `~/.agents/skills`. About page includes cloud agent, code review, CLI, app, VS Code and JetBrains agent mode.[2] | Common format plus invocation controls, subagents and dynamic context extensions. Native project/personal `.claude/skills`; enterprise and nested scopes also documented.[4] | Workspace `.cline/skills`, global `~/.cline/skills`; explicit invocation and enabled-by-default toggles documented.[8] |
| Always-on instructions | Repository-wide `.github/copilot-instructions.md`, path-specific files and agent instructions have surface-dependent support. Do not apply the GitHub.com hierarchy to every IDE.[3] | `CLAUDE.md` and related discovered memory files are concatenated into context, not overriding configuration files.[5] | Workspace/global rules combine, with workspace taking precedence for conflicts; toggles may disable rules.[9] |
| Name collision / precedence | Inspected docs establish instruction precedence on GitHub.com, not a universal skill-name collision order. Test duplicate personal/project skills before approval.[3] | Enterprise skill over personal over project for same-name collisions; distinct from memory-file concatenation.[4][5] | Global skill over project skill, but workspace rules over global rules. Do not conflate them.[8][9] |
| Trust and execution | GitHub warns shared skills are not verified, may contain injection or malicious scripts, and shell preapproval can remove a protective confirmation.[26] | Trust/permission dialogs and sandbox capability exist, but approved shell access can exceed working directory; unattended sessions differ.[7] | User toggles do not establish an enterprise security boundary. Current hook landing page redirects to SDK plugin documentation, which describes executable plugins.[8][10][23] |
| Hooks and enforcement | No universal Copilot hook parity established by inspected sources. Do not promise a portable hook API or use absence of research as proof none exists. | PreToolUse supports blocking, but command/http/MCP timeout does NOT block automatically; plain exit 1 is generally nonblocking without valid decision JSON. Agent SDK callbacks differ.[6] | SDK documents lifecycle hooks and explicit `fail_closed` policy. This is not proof every IDE/CLI installation exposes identical configuration or failure semantics.[23] |
| Distribution | Repository content can travel with code. `gh skill` public preview documents installation, source provenance, `@SHA` and `--pin`; pinned skills are skipped in updates.[26] | Repository/managed deployment plus client-specific facilities. Use an external immutable bundle manifest; do not presume Copilot's installer or plugin semantics apply unchanged.[4] | Commit workspace skill directories for team sharing. SDK plugins can come from npm/git/file sources; these require separate software-supply-chain review.[8][23] |
| Telemetry | Usage API spans multiple surfaces, but rich IDE detail requires telemetry; server-side active-user recovery does not fill missing dimensional data.[19] | OTel supports configured usage/cost/tool export; detailed skill names/tool arguments require explicit collection settings. Version-specific behavior is substantial.[20] | SDK lifecycle hooks provide instrumentation building blocks, not proof of a ready-made comparable enterprise skill dashboard or retention contract.[23] |
| Portable guarantee | Readable common content only; neither skill selection nor correct execution follows from installation.[3][26] | Shared format does not standardize extension behavior or guarantee instruction adherence.[4][5] | Shared content and client behavior remain different; toggles and name shadowing affect what executes.[8][9] |
### Version pinning, updates, rollback and stale laptops — recommended operational contract
These are recommended design choices, not features claimed to exist universally:
- Maintain one reviewed registry with stable skill IDs, human owner/reviewer, purpose, risk tier, provenance/license, immutable source revision, artifact digest, dependencies, supported client/OS/version range, evaluation revision/results, approval record, last review date, deprecation state, migration guide and rollback digest. Keep client-specific invocation and tool configuration outside the portable core, generated and tested as adapters.
- Release a manifest mapping stable IDs to exact immutable content and adapter revisions. Pin full commit/digest, not a mutable branch or tag alone. GitHub's preview installer provides pin/provenance mechanisms, but private Gitea compatibility and availability on Eldorado laptops have not been established.[26] A controlled vendored bundle is the simpler cross-client recommendation until proven otherwise.
- Promotion: author branch → independent review → format/resource validation → script/dependency/security checks → per-client behavioral and adversarial evals → canary → approval → distribution. Never permit a proposed skill to approve itself or edit its own promotion checks without independent review.
- Updates: compare signed/verified manifest and content digest; show release notes; stage atomically; restart/reload affected sessions where required by the client. Test whether already-open sessions retain old instructions. Do not infer reload from files changing.
- Rollback: restore a previously approved manifest AND adapter/dependency versions, revoke the unsafe revision centrally, expire affected sessions, rerun a smoke test and read back effective version. Reverting a file does not undo commands already executed or leaked data.
- Offline laptop: a verified cached bundle can support permitted low-risk local drafting, provided required local model/tool dependencies actually work offline. Local files being present do not mean cloud-backed inference is offline-capable. No claim of native offline inference or guaranteed durable telemetry buffering is made for any client.
- Stale/unsupported bundle: show a visible warning and prevent high-risk centrally mediated actions; allow explicitly approved low-risk work only. Recommended grace periods are risk decisions to settle, not fabricated vendor defaults. On reconnect, revalidate revocation and support status before obtaining privileged credentials or submitting gated delivery actions.
- Laptop evidence is fallible and may be altered by its owner. CI should verify repository-delivered manifest/content and actual output, but cannot prove which private prompt or skill a laptop truly used. Treat client attestations as advisory unless an independently verified device/runtime trust mechanism exists. Central enforcement must still protect work with no attestation.
- Avoid conflicting names and broad user-directory installation; prefix governed IDs and detect shadowing. Keep an explicit exception route with owner, scope, justification, expiry and compensating checks. An exception must not quietly disable protected-release controls.
## 4. Conflicts, weak evidence and what not to conclude
| Claim / conflict | Classification and resolution |
| --- | --- |
| “Project skills universally override user skills.” | Contradicted as a universal claim. Agent Skills implementation guide says this; current Claude and Cline product docs specify personal/global priority for skill collisions. Weight actual client documentation for that client, then verify behavior locally.[21][4][8] |
| “GitHub organization instructions enforce company policy everywhere.” | Unsupported. GitHub documents limited surfaces, personal/repository precedence, nondeterminism and head-branch review instructions. Use repository ownership and independently protected gates instead.[3] |
| “All clients support the same skills/hooks/telemetry.” | Unsupported. Standard content exists, but scope, extensions, hooks and instrumentation differ. Cline's hook page now points to SDK plugins; do not repeat historical hook file recipes without version testing.[1][10][23] |
| “Copilot supports JetBrains skills.” | About-skills page explicitly includes JetBrains; adding-skills overview currently omits it. This is documentation asymmetry, not sufficient evidence to declare unsupported. Verify the exact target host/version during adapter qualification.[2][26] |
| “Hook failure blocks dangerous work.” | Not generically true. Claude documents nonblocking timeout and exit-code cases; Cline SDK offers an explicit fail-closed setting. Test timeout, crash and malformed output separately.[6][23] |
| “AI makes developers 55.8% faster.” | Supported paraphrase only for Peng et al.'s controlled JavaScript HTTP-server task; abstract inspected. Not a software-organization forecast.[17] |
| “AI makes experienced developers 19% slower today.” | Misleading temporal/generalized claim. METR's early-2025 RCT involved 16 developers and 246 issues; the authors now label those results outdated as current capability evidence.[16] |
| “METR's newer results prove a universal speedup.” | Unsupported. Follow-up raw estimates suggest shorter task time, but intervals include no effect and authors warn participant/task selection and parallel-agent timing compromise the estimate.[22] |
| “Independent study proves a 21% Eldorado improvement.” | Unsupported. Paradis et al.'s abstract reports approximately 21% shorter task time for 96 Google engineers, with a large interval and explicit ecosystem/generalization caveats. Primary abstract, not full methods, inspected.[18] |
| “Agent says done / skill was loaded, therefore compliant.” | Unsupported. Outcome must be inspected in the environment; selection, reading, following, tool execution and outcome are separate observations.[11] |
Research hygiene correction: the initial URL seed for arXiv `2410.12944` was mistakenly labelled as a Cui field-experiments source. The retrieved body is Paradis et al.'s Google RCT; `sources.json` records the correct identity. No Cui statistic is used. Likewise, ACM SPACE returned HTTP 403; an author-hosted Microsoft Research publication summary was retrieved instead. These are bounded acquisition limitations, not invented evidence.
## 5. Target coordination, authority and organization mapping
Recommended trust flow: people and approved work records → one logical SDLC coordinator → task-scoped local-agent work proposals → repository review → independently administered CI/artifact evidence → authorized human release decision → release service. The coordinator should maintain workflow state, dependencies, requested evidence, retries and escalation. It should not be sole author, evaluator and approver, nor hold unrestricted production credentials.
Use durable IDs, idempotent task transitions and bounded retry budgets so multiple coordinator replicas cannot publish twice. Keep a human takeover and deterministic queue/authorization path if inference is unavailable. Project-level context sharing must be deliberately scoped; shared coordination is not permission to mix secrets, HR records, customer disputes and source code.
| Existing owner | Recommended accountability, without creating teams |
| --- | --- |
| Product Management | Choose outcome hypotheses and eligible work; accept product value and prioritize pilots rather than maximize AI usage. |
| Product Design | Validate usability/accessibility and customer research handling; review generated experience changes and design-task skills. |
| Product Engineering | Own code, practical skill authoring and local-agent use; review generated changes and remain responsible for maintainability. |
| Platform & IT | Own bundle delivery, device/client support, identity, workload isolation, central CI/release integration, telemetry transport and emergency revocation. |
| Architecture & Security | Own trust boundaries, data/permission risk classification, supply-chain approval requirements and security exceptions; review privileged skill proposals. |
| Quality Engineering | Own independent regression/adversarial suites, quality rubrics, gate evidence, failure sampling and evaluation reproducibility. |
| Data & AI | Coordinate the skills catalog and logical SDLC-agent evaluation with the documented AI Architect function; own statistical analysis and model/client experiment design. |
| Operations Squad | Supply sanitized service cases and operational acceptance; own human decisions on disputes, refunds, fraud and customer communications, not the coordinator. |
| Marketing Squad | Own brand/claim review, campaign approval and content rights; start with drafting/review skills rather than autonomous spending or publication. |
| Finance & Accounting Squad | Validate cost allocation and investment evidence; retain payout, accounting and financial approval segregation. |
| People & Administration Squad | Own training, onboarding, acceptable use and privacy-sensitive adoption feedback, with Platform & IT on access lifecycle; no individual developer ranking. |
Department leaders retain business accountability. Staffing capacity, named approvers and deputies need human confirmation; the operating-model role description is not evidence that every position is filled. For synthesis, map recommendations to existing responsibilities rather than publish uncertain staffing as fact.
### Threat and failure controls
NIST supplies a voluntary lifecycle governance framework and GenAI companion, not certification; the RMF core organizes work around govern, map, measure and manage.[13][14][25]
OWASP identifies external-content injection and recommends least privilege, human approval and adversarial testing.[15]
Recommended controls by failure mode:
- Malicious skill, script, dependency, MCP server or changed upstream tag: allowlisted source, immutable digest, independent review, license/provenance inventory, restricted network/runtime and no automatic arbitrary installs. A reviewed Markdown file does not make referenced executable code safe.
- Prompt injection from issues, repository instructions, web pages or tool results: treat as untrusted input; separate data from authority; limit credentials and egress; require independent authorization. GitHub code review's use of PR-head instructions is a concrete reason not to trust AI review as the sole gate.[3][15]
- Fabricated success, omitted tests, cherry-picked evidence or judge manipulation: verify actual artifact/CI state tied to exact commit and policy revision; keep evaluation checks outside candidate-controlled changes; sample human review.
- Overbroad local shell permissions: isolate process/OS identity, filesystem, network and secrets. A profile, skill folder or current directory is not containment. Claude's own docs distinguish permission prompts from sandbox controls.[7]
- Duplicate or looping agent work, runaway spend and unavailable coordinator: task idempotency, bounded retries/time/cost, lease expiry, human recovery and read-only/degraded mode. Never downgrade a release gate because the model or evidence service is unavailable.
- Model/client/context drift or stale session instructions: version-qualified adapters; requalification on material updates; compaction/long-session tests; version-aware incident analysis.
- Privacy leakage through telemetry or shared memory: minimal structured events, pseudonymous IDs, restricted retention and access; no raw prompts, secrets or source contents by default. Explicitly review the privacy cost before enabling detailed tool/skill-name export.[20]
- Kill switch: independently revoke mutation/release credentials and stop scheduling; disable affected revisions and preserve evidence. Sending “stop” to a model is not the security boundary.
## 6. Skill quality, evaluation and lifecycle recommendations
A useful skill must state when to use it, when not to use it, prerequisites, exact steps, expected artifacts, verification, failure handling and escalation. Keep the portable body small and reference bounded resources. The format specification supports progressive disclosure and recommends concise main files.[1]
Proposed evaluation layers:
1. Static contract: parse metadata; verify stable identity/owner/review date; validate links and resource paths; detect embedded secrets, prohibited dependencies, unsafe commands and unsupported client-only fields.
2. Selection tests: positive trigger tasks, near misses and negative tasks; report false activation and missed activation separately. A larger catalog should be tested for routing regressions, not presumed better.
3. Execution tests: run representative sanitized tasks in disposable environments with fixed starting repositories; verify real outputs, not self-reports. Include missing prerequisite, invalid input, tool timeout, denied permission, malformed evidence and interrupted session.
4. Safety tests: injected repository text, malicious referenced scripts, conflicting global/project skills, user toggle, unsupported version, expired/revoked bundle, offline network, context compaction and stale in-memory version.
5. Outcome grading: deterministic tests for mechanically checkable constraints; independent human review for product/design/maintainability; model judges only with explicit rubric and human calibration. Record disagreements and judge version.[11]
6. Comparison: same tasks and conditions for no-skill/current-approved/candidate versions; repeated trials, failure/abandonment retained; separate client/model/skill changes. Report sample size, variability, confidence and task mix, not only a mean pass percentage.[11]
Recommended promotion policy: all mandatory deterministic safety/authorization checks must pass; any observed critical unauthorized action blocks promotion pending investigation. This is a proposed policy, NOT proof of zero residual risk. Quantitative success-rate thresholds should be set after pilot variance and business impact are known, by risk tier; do not invent universally validated percentages. Privileged release/payment/customer-impact workflows need stronger review than read-only documentation assistance.
Lifecycle: proposed → experimental → approved → deprecated → retired, with quarantine available from every active state. Review triggered by incident, tool/model/client change, dependency change, usage feedback or scheduled owner review. Each approved skill needs an owner and backup, change history, support window, migration route and known-good rollback. Deprecation should warn, offer replacement and track consumers before retirement; urgent security revocation overrides convenience. Do not retain unowned skills merely because they remain popular.
Adoption should begin with training on how to inspect results, report failures and refuse unsafe requests. Build examples from actual team workflows; offer office hours and opt-in feedback; record why a skill was rejected. Measure whether people can perform and verify work, not whether they obey a target number of invocations. Avoid automatically promoting agent-written corrections into the governed catalog.
## 7. KPI instrumentation options and recommended choices
### Practical source tiers
A. Start with authoritative repository/CI/deployment/incident events. These can measure delivery outcomes without observing every laptop prompt. No event feed has been inspected here; feasibility must be verified with the current hosting platforms and retention.
B. Add a governed catalog/version inventory and opt-in task-level skill/client declarations. This supplies adoption and support coverage, but declarations are not proof of execution or adherence.
C. Add client adapters where supported: Copilot usage APIs and Claude OTel may help; their fields, licensing, enabled state and missingness differ.[19][20] Cline SDK hooks are an instrumentation option requiring engineering and exact-version qualification, not a proven universal collector.[23]
D. Use short anonymous surveys, work sampling and pilot comparisons for cognitive load, confidence and rework not visible in logs. SPACE's multidimensional premise supports this combination; surveys alone cannot establish time saved.[24][16]
### Candidate measurement contracts for synthesis
These are recommended options, not observed metrics or approved targets. All denominators must use a predeclared eligible population and reporting window; zero denominators produce “not available,” never zero performance.
| Measure | Definition and useful source | Owner / interpretation / missingness |
| --- | --- | --- |
| Validated task success | Independently accepted successful tasks / all eligible initiated tasks with matured observation window; count failed/abandoned tasks as nonsuccess and show still-open separately. Task ID + acceptance evidence, not agent message. | Quality Engineering; weekly pilot report. Show task type, sample size and unobserved outcomes separately; exclude cancelled-before-start work by stated rule, not post-hoc cherry-picking. |
| Skill selection precision / recall | Correct relevant activations / all activations; relevant tasks with activation / all relevant tasks, on a labeled evaluation set. | Data & AI with Quality Engineering; per candidate/client release. Not obtainable from skill-loaded logs without relevance labels. |
| Supported bundle coverage | Eligible active installations with verified supported revision / eligible active installations inventoried. | Platform & IT; weekly. Unknown/invisible laptops remain unknown and inventory coverage is separately reported; do not treat silence as compliant. |
| Useful adoption | Distinct eligible participants using an approved skill in an eligible workflow / eligible participants during window, plus qualitative usefulness. | Data & AI and People & Administration Squad; monthly aggregate. Separate declared, observed and inferred usage; suppress small cohorts and never rank people. |
| Change lead time | Commit-to-production elapsed time per deployed change; report distribution and population. Repository + deployment SHA association. | Platform & IT / Product Engineering; service-level trend. Missing SHA mapping is missing coverage, not instantaneous delivery. DORA definition, not PR-open-to-merge time.[12] |
| Deployment frequency | Production deployments / stated period, service-scoped. | Platform & IT; report with quality measures. Exclude retries/duplicate events; no incentive to split meaningless deployments.[12] |
| Change fail rate | Production deployments requiring immediate intervention / all production deployments. | Quality Engineering / Platform & IT; require incident/rollback linkage and intervention definition. Missing incident links limit confidence.[12] |
| Failed deployment recovery time | Recovery timestamp minus failed-deployment failure timestamp; distribution over relevant failed deployments. | Platform & IT; unresolved failures shown separately, never dropped as if recovered. Not all-incident MTTR.[12] |
| Deployment rework rate | Unplanned incident-driven production deployments / all production deployments. | Platform & IT; preserve planned/unplanned classification and incident link. DORA fifth metric, distinct from change fail rate.[12] |
| Escaped quality issues | Severity-stratified post-acceptance defects / accepted changes with completed follow-up, plus absolute incident counts. | Quality Engineering; agreed maturation window. Do not compare immature and mature cohorts or imply every defect was AI-caused. |
| Safety / policy violations | Confirmed prohibited actions or attempted bypasses / evaluated eligible runs, with severe incidents also as absolute counts. | Architecture & Security; immediate incident response and periodic trend. Separate detected attempts, blocked attempts and actual effects; zero observed is not proof of no risk. |
| Cost per accepted outcome | Allocated license + model/tool/compute + human review/rework costs / independently accepted outcomes; include costs of failed/abandoned attempts in numerator. | Finance & Accounting Squad; monthly. Show uncertain allocation and missing spend; do not call model tokens the whole cost. |
| Human experience | Aggregate survey distribution for cognitive load, confidence in verification and perceived usefulness, with response count / invited eligible population. | People & Administration Squad / Data & AI; periodic. Self-selection and perception bias explicit; no employee surveillance dashboard. |
| Maintenance health | Approved skills within review window / active approved skills; separately count ownerless, deprecated-in-use and revoked-in-use revisions. | Data & AI / Platform & IT; catalog release and periodic review. Review stamp alone is not quality evidence. |
Recommended minimal event schema: `event_id`, `schema_version`, `event_type`, `occurred_at`, `received_at`, `producer`, `task_id`, `attempt_id`, `project_id`, `pseudonymous_actor_id` where necessary, `client/host/version`, `model_id/version_if_exposed`, `skill_id/revision/digest`, `adapter_revision`, `policy_revision`, `repo_commit`, `outcome`, `reason_code`, `evidence_ref`, `cost_amount/currency/basis`, `measurement_origin` (server/client/self-report), and `consent_or_collection_basis`. Fields can be nullable with reason; do not fabricate a model revision hidden by a provider.
Recommended ingestion rules: deduplicate by producer + event_id; model retries as attempts, not extra completed tasks; preserve late events and recompute versioned aggregates; use server receipt and skew checks rather than trusting laptop clocks blindly. A lost terminal event is unknown until reconciled, not automatically success or abandonment. Distinguish paused, failed, abandoned, cancelled-before-start and accepted. A skill-loaded event records loading only. A task can involve multiple skills/versions and agents; avoid attributing the entire outcome or labor time to every invocation.
Suggested privacy starting point for human approval: short raw structured-event retention (for example 30 days), longer aggregate trends (for example 12 months), no raw prompts/code/screens by default, role-limited incident evidence, and explicit deletion/access process. These durations are policy proposals, not legal requirements or existing Eldorado settings. Confirm employment/privacy constraints with responsible owners before collection.
Pilot comparison recommendation: define task classes and quality acceptance first; collect a prospective baseline and eligible population; compare matched or randomized cohorts when ethical/practical; record concurrent process/client/model changes. Keep quality and risk guardrails fixed. Report descriptive trends separately from causal estimates. Measure elapsed lead time and human effort separately because parallel agents break the assumption that elapsed time equals labor saved; METR's follow-up explicitly raises this issue.[22]
For executable downstream examples, use clearly synthetic fixtures covering duplicates, retries, unknown outcomes, multiple skills, zero denominator, immature quality window and late deployment/incident events. Assert exact expected numerator/denominator and units. Do not display synthetic values as Eldorado baseline data.
## 8. Recommended sequence and evidence gates
- Foundation: inventory actual clients, repositories, data sensitivity, available events and accountable owners. Exit only with a supported pilot boundary and explicit forbidden actions, not with a catalog full of untested skills.
- Low-risk pilot: small set of useful read-only/drafting/test-assistance skills; reviewed immutable bundles; exact-client smoke and adversarial tests; human review and failure reporting. Exit with reproducible quality evidence and usable measurement coverage.
- Governed distribution: add version-aware adapters, onboarding, staged upgrades, rollback/revocation drills and exception expiry. Exit when stale/offline/unsupported clients cannot bypass central privileged gates.
- Logical SDLC coordination: coordinator starts by observing and assembling evidence, then proposing work and routing approvals. Expand mutation only after idempotency, separation of duties, scoped credentials, outage recovery and kill-switch tests pass.
- Broader departmental use: Operations, Marketing, Finance & Accounting and HR & Administration adopt domain-specific skills with their own human decisions, data boundaries and outcome measures. Do not copy coding autonomy into payments, personnel decisions or disputes.
No invented rollout dates, budget, staffing capacity or required improvement percentage. Scale only when quality, risk and cost evidence support it; stop or narrow scope when benefits are not demonstrated.
## 9. Source coverage and unresolved questions
Coverage: 26 retrieved primary-source records with 50 literal evidence passages. Includes Agent Skills specification and integration guidance; GitHub support/instructions/install/metrics; Claude skills/memory/hooks/security/monitoring; Cline skills/rules/hooks and linked SDK plugins; agent-evaluation methodology; current DORA definitions; author-hosted SPACE summary; NIST framework/core/GenAI profile landing page; OWASP prompt injection; two primary research abstracts plus METR historical study and 2026 follow-up. This is substantive source-body review, but not a systematic literature review, independent replication, legal assessment or live client qualification. Full PDFs for the two arXiv papers and NIST GenAI profile were not reviewed; claims are limited accordingly. ACM's 403 is recorded and resolved only to the author-hosted summary, not bypassed or silently called full text.
Open decisions/evidence needed before implementation:
1. Which actual client hosts, versions, models, OSs, subscriptions and managed-device controls are in use? What is the allowed support window?
2. Which repository/CI/release systems and protected-branch controls are authoritative? Can required checks and policy definitions be altered by a candidate PR or bypassed by administrators?
3. Who fills each approving/operating function and who provides cover? What pilot capacity is available? The documented organization does not establish personnel availability.
4. What data may reach each model provider or telemetry endpoint; what retention/training/region and contractual terms apply? No provider contract was inspected.
5. Which low-risk workflows have enough representative tasks and independent quality labels? What is the actual eligible population and baseline coverage?
6. What timeout, stale-bundle grace, critical-risk rejection and emergency-revocation policies will owners approve? Can laptop access be independently revoked?
7. Does the chosen distributor support private Gitea securely, and can every supported client report effective skill/adapter versions without collecting sensitive content?
8. What causal claims, if any, can the proposed pilot design justify given self-selection, learning, task mix and parallel work?
9. Which detailed Cline/Copilot hooks and offline/resume behavior exist in the precise deployed builds? Sources here do not establish cross-client equivalence.
Synthesis handoff: keep this research trail outside reader-facing portal prose. Use ordinary operating-model language for the strategy, preserve all existing teams and leadership uncertainty, visibly distinguish target maturity from current state, and retain recommendation/source mapping internally. No portal publication is part of this research card.
## Sources
[1] https://agentskills.io/specification — Agent Skills specification
[2] https://docs.github.com/en/copilot/concepts/agents/about-agent-skills — Copilot Agent Skills
[3] https://docs.github.com/en/copilot/concepts/prompting/response-customization — Copilot custom instructions
[4] https://code.claude.com/docs/en/skills — Claude Code skills
[5] https://code.claude.com/docs/en/memory — Claude Code memory
[6] https://code.claude.com/docs/en/hooks — Claude Code hooks
[7] https://code.claude.com/docs/en/security — Claude Code security
[8] https://docs.cline.bot/customization/skills — Cline skills
[9] https://docs.cline.bot/customization/cline-rules — Cline rules
[10] https://docs.cline.bot/customization/hooks — Cline hooks
[11] https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents — Demystifying evals for AI agents
[12] https://dora.dev/guides/dora-metrics — DORA software delivery metrics
[13] https://www.nist.gov/itl/ai-risk-management-framework — NIST AI RMF
[14] https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence — NIST Generative AI Profile
[15] https://genai.owasp.org/llmrisk/llm01-prompt-injection — OWASP prompt injection
[16] https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study — Experienced developer RCT
[17] https://arxiv.org/abs/2302.06590 — AI developer productivity experiment
[18] https://arxiv.org/abs/2410.12944 — Field experiments in software development
[19] https://docs.github.com/en/copilot/concepts/billing-and-usage/copilot-usage-metrics/copilot-metrics — Copilot metrics concepts
[20] https://code.claude.com/docs/en/monitoring-usage — Claude Code monitoring
[21] https://agentskills.io/client-implementation/adding-skills-support — Agent Skills security
[22] https://metr.org/blog/2026-02-24-uplift-update — METR productivity update 2026
[23] https://docs.cline.bot/sdk/plugins — Cline SDK plugins
[24] https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity-theres-more-to-it-than-you-think — SPACE research publication
[25] https://airc.nist.gov/airmf-resources/airmf/5-sec-core — NIST AI RMF Core
[26] https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-cloud-agent/add-skills — Copilot adding skills
+564
View File
@@ -0,0 +1,564 @@
[
{
"id": 1,
"title": "Agent Skills specification",
"url": "https://agentskills.io/specification",
"publisher": "Agent Skills",
"retrieved_at": "2026-10-04T10:27:53.189577+00:00",
"claims": [
{
"claim": "The specification defines a skill directory with SKILL.md, not a universal deployment or enforcement service.",
"evidence": "A skill is a directory containing, at minimum, a SKILL.md file:",
"limitations": "Format compatibility does not certify discovery, precedence, tool permissions, safety, or instruction compliance."
},
{
"claim": "allowed-tools is experimental in the common specification.",
"evidence": "allowed-toolsNoSpace-separated string of pre-approved tools the skill may use. (Experimental)",
"limitations": "Format compatibility does not certify discovery, precedence, tool permissions, safety, or instruction compliance."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/01.txt",
"body_sha256": "0913e84055bc7eb05649aa49123a783dc1a413e4c31d0861f9ac367849c7b00e",
"evidence_type": "primary source body"
},
{
"id": 2,
"title": "Copilot Agent Skills",
"url": "https://docs.github.com/en/copilot/concepts/agents/about-agent-skills",
"publisher": "GitHub",
"retrieved_at": "2026-10-04T10:27:53.333904+00:00",
"claims": [
{
"claim": "GitHub documents skill support across specified Copilot surfaces, not all modes indiscriminately.",
"evidence": "Agent skills work with Copilot cloud agent, Copilot code review, the GitHub Copilot CLI, the GitHub Copilot app, and agent mode in Visual Studio Code and JetBrains IDEs.",
"limitations": "Vendor documentation, not an Eldorado installation test. Separate adding-skills page omits JetBrains in its overview; validate exact host/version."
},
{
"claim": "Copilot documents multiple project skill paths.",
"evidence": "Project skills, stored in your repository (.github/skills, .claude/skills, or .agents/skills)",
"limitations": "Vendor documentation, not an Eldorado installation test. Separate adding-skills page omits JetBrains in its overview; validate exact host/version."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/02.txt",
"body_sha256": "26c86bc2afa4813c46361429f015f687164a69f39030fa05dbc1f513a04c0ca3",
"evidence_type": "primary source body"
},
{
"id": 3,
"title": "Copilot custom instructions",
"url": "https://docs.github.com/en/copilot/concepts/prompting/response-customization",
"publisher": "GitHub",
"retrieved_at": "2026-10-04T10:27:53.960265+00:00",
"claims": [
{
"claim": "GitHub explicitly warns that custom-instruction following is nondeterministic.",
"evidence": "Due to the non-deterministic nature of AI, Copilot may not always follow your custom instructions in exactly the same way every time they are used.",
"limitations": "Instruction precedence is surface-specific and is not enforceable authorization. Head-branch instructions create a review trust risk."
},
{
"claim": "The GitHub.com instruction hierarchy prioritizes personal over repository over organization instructions.",
"evidence": "Multiple types of custom instructions can apply to a request sent to Copilot. Personal instructions take the highest priority. Repository instructions come next, and then organization instructions are prioritized last. However, all sets of relevant instructions are provided to Copilot.",
"limitations": "Instruction precedence is surface-specific and is not enforceable authorization. Head-branch instructions create a review trust risk."
},
{
"claim": "Copilot review reads instructions and skills from the PR head branch.",
"evidence": "When reviewing a pull request, Copilot reads repository custom instructions, agent instructions, and agent skills from the head branch (the branch with your changes), not the base branch. For example, when merging my-feature-branch into main, Copilot uses the instructions and skills in my-feature-branch, so you can test changes to them in the same pull request without merging them first.",
"limitations": "Instruction precedence is surface-specific and is not enforceable authorization. Head-branch instructions create a review trust risk."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/03.txt",
"body_sha256": "1c2c190052653b8baad1478dc53b7a0657b741be0dbb0ef202b3f503c6b3379c",
"evidence_type": "primary source body"
},
{
"id": 4,
"title": "Claude Code skills",
"url": "https://code.claude.com/docs/en/skills",
"publisher": "Anthropic",
"retrieved_at": "2026-10-04T10:27:53.088978+00:00",
"claims": [
{
"claim": "Claude skill name collisions prioritize enterprise over personal over project.",
"evidence": "Two of enterprise, personal, and projectEnterprise over personal, and personal over project. With deploy in both ~/.claude/skills/ and the project’s .claude/skills/, /deploy runs the personal one",
"limitations": "Vendor extensions and changing versions prevent assuming cross-client parity. No local client exercised."
},
{
"claim": "Claude extends the common skill format with client-specific functionality.",
"evidence": "Claude Code skills follow the Agent Skills open standard, which works across multiple AI tools. Claude Code extends the standard with additional features like invocation control, subagent execution, and dynamic context injection. See Using skill frontmatter outside Claude Code for which frontmatter fields are part of the standard and which are Claude Code extensions.",
"limitations": "Vendor extensions and changing versions prevent assuming cross-client parity. No local client exercised."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/04.txt",
"body_sha256": "4ac3de187f2ba050da149e1bea3364a959366be03963e7a76dc9f9041bdb4e1f",
"evidence_type": "primary source body"
},
{
"id": 5,
"title": "Claude Code memory",
"url": "https://code.claude.com/docs/en/memory",
"publisher": "Anthropic",
"retrieved_at": "2026-10-04T10:27:52.994293+00:00",
"claims": [
{
"claim": "Claude memory instructions are context, not enforced configuration.",
"evidence": "Claude Code has two complementary memory systems. Both are loaded at the start of every conversation. Claude treats them as context, not enforced configuration. To block an action regardless of what Claude decides, use a PreToolUse hook instead. The more specific and concise your instructions, the more consistently Claude follows them.",
"limitations": "Context order does not establish deterministic behavior; conflicting prose may still produce arbitrary choices."
},
{
"claim": "Claude concatenates discovered memory files rather than treating them as override-only configuration.",
"evidence": "All discovered files are concatenated into context rather than overriding each other. Across the directory tree, content is ordered from the filesystem root down to your working directory. For the foo/bar/ example, foo/CLAUDE.md appears in context before foo/bar/CLAUDE.md, so instructions closer to where you launched Claude are read last. Within each directory, CLAUDE.local.md is appended after CLAUDE.md, so your personal notes are the last thing Claude reads at that level.",
"limitations": "Context order does not establish deterministic behavior; conflicting prose may still produce arbitrary choices."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/05.txt",
"body_sha256": "b19869f25886f95f8d93cec941482cbf9472b507ab8d643b446b1450b915e0df",
"evidence_type": "primary source body"
},
{
"id": 6,
"title": "Claude Code hooks",
"url": "https://code.claude.com/docs/en/hooks",
"publisher": "Anthropic",
"retrieved_at": "2026-10-04T10:27:53.059984+00:00",
"claims": [
{
"claim": "Claude PreToolUse command/http/MCP hook timeout is not fail-closed.",
"evidence": "A timed-out command, http, or mcp_tool hook doesn’t block the tool call. The call continues through the normal permission flow, so don’t count on a stalled hook to act as a gate.",
"limitations": "Event and hook-family distinctions matter; no universal fail-closed promise. No hook was installed or tested here."
},
{
"claim": "A conventional exit code 1 alone does not reliably block most Claude hook events.",
"evidence": "For most hook events, exit code 2 is the only exit code that blocks through the code alone. Without valid JSON on stdout, Claude Code treats exit code 1 as a non-blocking error and proceeds with the action, even though 1 is the conventional Unix failure code. If your hook is meant to enforce a policy, use exit 2. The worktree events differ: any non-zero exit code from WorktreeCreate aborts worktree creation, and any non-zero exit code from WorktreeRemove makes worktree removal fail if the directory still exists afterward.",
"limitations": "Event and hook-family distinctions matter; no universal fail-closed promise. No hook was installed or tested here."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/06.txt",
"body_sha256": "ae1919a5b06aa6d4a213bfafcaab1d2473f0b89d53b9fcc35358a19ab9ca3e5d",
"evidence_type": "primary source body"
},
{
"id": 7,
"title": "Claude Code security",
"url": "https://code.claude.com/docs/en/security",
"publisher": "Anthropic",
"retrieved_at": "2026-10-04T10:27:52.953904+00:00",
"claims": [
{
"claim": "Claude working-directory approval is not OS-level containment.",
"evidence": "Working directory boundary: In Manual mode, Claude Code asks you before its file tools read or write outside the folder where it was started and its subfolders. The boundary is a permission prompt, so a Bash command you approve can still write anywhere your user account can",
"limitations": "Vendor capability is not evidence the sandbox is enabled; approved shell commands inherit actual OS authority."
},
{
"claim": "OS-level Bash restrictions require sandboxing.",
"evidence": "To restrict Bash commands at the operating system level, turn on sandboxing",
"limitations": "Vendor capability is not evidence the sandbox is enabled; approved shell commands inherit actual OS authority."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/07.txt",
"body_sha256": "a8a2ad7d077347f1d3ea43549b0be1070e03c7737647f73ef40035d5dafb5f40",
"evidence_type": "primary source body"
},
{
"id": 8,
"title": "Cline skills",
"url": "https://docs.cline.bot/customization/skills",
"publisher": "Cline",
"retrieved_at": "2026-10-04T10:27:53.250798+00:00",
"claims": [
{
"claim": "Cline global skill overrides same-name project skill.",
"evidence": "When a global skill and project skill have the same name, the global skill takes precedence. This lets you keep general-purpose skills globally while using project-specific skills in .cline/skills/ so the whole team can use them.",
"limitations": "Current docs disagree with Agent Skills guide generalization. Exact installed Cline version remains unknown."
},
{
"claim": "Discovered Cline skills have user-controlled enable/disable toggles.",
"evidence": "Every skill has a toggle to enable or disable it. This lets you control which skills are active without deleting the skill directory. Skills are enabled by default when discovered.",
"limitations": "Current docs disagree with Agent Skills guide generalization. Exact installed Cline version remains unknown."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/08.txt",
"body_sha256": "38e6af22c7e9f78e90e899a7bd285e6556bfd64197fbc3f7cf279cc9ff29d416",
"evidence_type": "primary source body"
},
{
"id": 9,
"title": "Cline rules",
"url": "https://docs.cline.bot/customization/cline-rules",
"publisher": "Cline",
"retrieved_at": "2026-10-04T10:27:53.308915+00:00",
"claims": [
{
"claim": "Cline workspace rules override conflicting global rules, unlike skill name precedence.",
"evidence": "When both workspace and global rules exist, Cline combines them. Workspace rules take precedence when they conflict with global rules. See Storage Locations for more guidance.",
"limitations": "Rule precedence is not skill precedence; both are guidance and can be toggled."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/09.txt",
"body_sha256": "c1a2353c024491a42961e5ea171c341c66bbed2647464d200bf90fdbaa8a48e3",
"evidence_type": "primary source body"
},
{
"id": 10,
"title": "Cline hooks",
"url": "https://docs.cline.bot/customization/hooks",
"publisher": "Cline",
"retrieved_at": "2026-10-04T10:27:53.370475+00:00",
"claims": [
{
"claim": "Cline hooks landing page delegates details to SDK plugins.",
"evidence": "See details under SDK Plugins page.",
"limitations": "Landing page alone proves no event schema or IDE compatibility. SDK page was followed separately."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/10.txt",
"body_sha256": "2ebd8cd1c0b11fec466a3b577971382d2c4ec1f9911679b10100029e9b3be095",
"evidence_type": "primary source body"
},
{
"id": 11,
"title": "Demystifying evals for AI agents",
"url": "https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents",
"publisher": "Anthropic",
"retrieved_at": "2026-10-04T10:27:53.451618+00:00",
"claims": [
{
"claim": "Repeated trials are needed for variable agent outputs.",
"evidence": "Each attempt at a task is a trial. Because model outputs vary between runs, we run multiple trials to produce more consistent results.",
"limitations": "Vendor-authored methodology and examples, not independent proof of any skill gain or target threshold."
},
{
"claim": "Agent outcome evaluation must inspect environment state rather than trust claimed completion.",
"evidence": "The outcome is the final state in the environment at the end of the trial. A flight-booking agent might say “Your flight has been booked” at the end of the transcript, but the outcome is whether a reservation exists in the environment’s SQL database.",
"limitations": "Vendor-authored methodology and examples, not independent proof of any skill gain or target threshold."
},
{
"claim": "Agent evals combine code, model and human graders.",
"evidence": "Agent evaluations typically combine three types of graders: code-based, model-based, and human. Each grader evaluates some portion of either the transcript or the outcome. An essential component of effective evaluation design is to choose the right graders for the job.",
"limitations": "Vendor-authored methodology and examples, not independent proof of any skill gain or target threshold."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/11.txt",
"body_sha256": "7940a33d8e19ff68aac5e3c095e0fa771565a4b2416e8a781e9ed02d55668a12",
"evidence_type": "primary source body"
},
{
"id": 12,
"title": "DORA software delivery metrics",
"url": "https://dora.dev/guides/dora-metrics/",
"publisher": "DORA",
"retrieved_at": "2026-10-04T10:27:53.335018+00:00",
"claims": [
{
"claim": "DORA now documents five software-delivery performance metrics.",
"evidence": "These metrics have evolved alongside the technology landscape: shifting from the original four keys to the current five-metric model. To understand the research behind changes like the move from MTTR to Failed Deployment Recovery Time, read A history of DORA’s software delivery metrics.",
"limitations": "Service/team outcome measures, not causal AI-effect estimates or individual productivity scores."
},
{
"claim": "DORA change lead time runs from commit to production deployment.",
"evidence": "Change lead time: The amount of time it takes for a change to go from committed to version control to deployed in production.",
"limitations": "Service/team outcome measures, not causal AI-effect estimates or individual productivity scores."
},
{
"claim": "DORA deployment rework measures unplanned incident-driven deployments.",
"evidence": "Deployment rework rate: The ratio of deployments that are unplanned but happen as a result of an incident in production.",
"limitations": "Service/team outcome measures, not causal AI-effect estimates or individual productivity scores."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/12.txt",
"body_sha256": "0535efe22d238f38d1de5ec655efeb44a313e59f8a71085e902bde5c7a693cb9",
"evidence_type": "primary source body"
},
{
"id": 13,
"title": "NIST AI RMF",
"url": "https://www.nist.gov/itl/ai-risk-management-framework",
"publisher": "NIST",
"retrieved_at": "2026-10-04T10:27:53.731461+00:00",
"claims": [
{
"claim": "NIST AI RMF is voluntary guidance for trustworthiness across the AI lifecycle.",
"evidence": "Led by the Information Technology Laboratory (ITL) AI Program, and in collaboration with the private and public sectors, NIST has developed a framework to better manage risks to individuals, organizations, and society associated with artificial intelligence (AI). The NIST AI Risk Management Framework (AI RMF) is intended for voluntary use and to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems.",
"limitations": "Voluntary framework, not certification, legal advice, or evidence of Eldorado compliance."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/14.txt",
"body_sha256": "2fed4e9811f05579384a23b9c3b81600ddf0f011fc4cfe638352a817f91ed603",
"evidence_type": "primary source body"
},
{
"id": 14,
"title": "NIST Generative AI Profile",
"url": "https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence",
"publisher": "NIST",
"retrieved_at": "2026-10-04T10:27:53.743664+00:00",
"claims": [
{
"claim": "NIST GenAI profile is a cross-sectoral companion to AI RMF.",
"evidence": "This document is a cross-sectoral profile of and companion resource for the AI Risk Management Framework (AI RMF 1.0) for Generative AI, pursuant to President Biden's Executive Order (EO) 14110 on Safe, Secure, and Trustworthy Artificial Intelligence. The AI RMF was released in January 2023, and is intended for voluntary use and to improve the ability of organizations to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems.",
"limitations": "Publication landing-page abstract inspected; full PDF not read. Do not attribute detailed risk taxonomy to this excerpt."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/15.txt",
"body_sha256": "7998580f4dc83770568dc361ecae1d46e1d652a2df12a278263b88de6cbfb624",
"evidence_type": "primary source body"
},
{
"id": 15,
"title": "OWASP prompt injection",
"url": "https://genai.owasp.org/llmrisk/llm01-prompt-injection/",
"publisher": "OWASP",
"retrieved_at": "2026-10-04T10:27:53.529810+00:00",
"claims": [
{
"claim": "External sources can carry prompt injections.",
"evidence": "Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files. The content may have in the external content data that when interpreted by the model, alters the behavior of the model in unintended or unexpected ways. Like direct injections, indirect injections can be either intentional or unintentional.",
"limitations": "Mitigations reduce exposure; no guarantee that prompt injection is eliminated."
},
{
"claim": "OWASP recommends least privilege.",
"evidence": "Provide the application with its own API tokens for extensible functionality, and handle these functions in code rather than providing them to the model. Restrict the model’s access privileges to the minimum necessary for its intended operations.",
"limitations": "Mitigations reduce exposure; no guarantee that prompt injection is eliminated."
},
{
"claim": "OWASP recommends human approval for high-risk operations.",
"evidence": "Implement human-in-the-loop controls for privileged operations to prevent unauthorized actions.",
"limitations": "Mitigations reduce exposure; no guarantee that prompt injection is eliminated."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/16.txt",
"body_sha256": "5bfcc42a422165d0a6b25f28fbff144fc0dec5289f1a672a40203e1d64200c07",
"evidence_type": "primary source body"
},
{
"id": 16,
"title": "Experienced developer RCT",
"url": "https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/",
"publisher": "METR",
"retrieved_at": "2026-10-04T10:27:54.289507+00:00",
"claims": [
{
"claim": "METR early-2025 RCT found 19% longer task time in its setting.",
"evidence": "We conduct a randomized controlled trial (RCT) to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower. We view this result as a snapshot of early-2025 AI capabilities in one relevant setting; as these systems continue to rapidly evolve, we plan on continuing to use this methodology to help estimate AI acceleration from AI R&D automation.",
"limitations": "Historical narrow setting, tools from early 2025; superseded as a current estimate by authors. No Eldorado forecast."
},
{
"claim": "Study involved 16 experienced developers and 246 issues on their repositories.",
"evidence": "To directly measure the real-world impact of AI tools on software development, we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors that would normally be part of their regular work. Then, we randomly assign each issue to either allow or disallow use of AI while working on the issue. When AI is allowed, developers can use any tools they choose (primarily Cursor Pro with Claude 3.5/3.7 Sonnet—frontier models at the time of the study); when disallowed, they work without generative AI assistance. Developers complete these tasks (which average two hours each) while recording their screens, then self-report the total implementation time they needed. We pay developers $150/hr as compensation for their participation in the study.",
"limitations": "Historical narrow setting, tools from early 2025; superseded as a current estimate by authors. No Eldorado forecast."
},
{
"claim": "METR now explicitly marks those historical results as not current.",
"evidence": "We have released results that are current as of early 2026, in a continuation of this study. We believe these historical results no longer reflect the current impact of AI models on open-source developer productivity.",
"limitations": "Historical narrow setting, tools from early 2025; superseded as a current estimate by authors. No Eldorado forecast."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/17.txt",
"body_sha256": "f9abab8a3ad312279800d8327122ebae70c13804d9f91fda1788db1a466a178b",
"evidence_type": "primary source body"
},
{
"id": 17,
"title": "AI developer productivity experiment",
"url": "https://arxiv.org/abs/2302.06590",
"publisher": "Peng et al.",
"retrieved_at": "2026-10-04T10:27:53.477297+00:00",
"claims": [
{
"claim": "Peng et al. controlled HTTP-server task study reports 55.8% faster completion with Copilot.",
"evidence": "Abstract:Generative AI tools hold promise to increase human productivity. This paper presents results from a controlled experiment with GitHub Copilot, an AI pair programmer. Recruited software developers were asked to implement an HTTP server in JavaScript as quickly as possible. The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group. Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.",
"limitations": "Primary paper abstract inspected, not full methods. Bounded JavaScript task; not whole-SDLC productivity."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/18.txt",
"body_sha256": "76c71ee9844c5066d184ec5006945c48959dfcd75854842463cd756f348a3140",
"evidence_type": "primary research abstract"
},
{
"id": 18,
"title": "How much does AI impact development speed? An enterprise-based randomized controlled trial",
"url": "https://arxiv.org/abs/2410.12944",
"publisher": "Paradis et al. / arXiv",
"retrieved_at": "2026-10-04T10:27:53.495104+00:00",
"claims": [
{
"claim": "Paradis et al. Google RCT abstract reports about 21% shorter task time with a large confidence interval, among 96 engineers.",
"evidence": "Abstract:How much does AI assistance impact developer productivity? To date, the software engineering literature has provided a range of answers, targeting a diversity of outcomes: from perceived productivity to speed on task and developer throughput. Our randomized controlled trial with 96 full-time Google software engineers contributes to this literature by sharing an estimate of the impact of three AI features on the time developers spent on a complex, enterprise-grade task. We found that AI significantly shortened the time developers spent on task. Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21\\%, although our confidence interval is large. We also found an interesting effect whereby developers who spend more hours on code-related activities per day were faster with AI. Product and future research considerations are discussed. In particular, we invite further research that explores the impact of AI at the ecosystem level and across multiple suites of AI-enhanced tools, since we cannot assume that the effect size obtained in our lab study will necessarily apply more broadly, or that the effect of AI found using internal Google tooling in the summer of 2024 will translate across tools and over time.",
"limitations": "Primary paper abstract inspected, not full methods. Google-specific task/tool setting; no universal gain claim. Retrieval seed title was wrong and corrected from body."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/19.txt",
"body_sha256": "e3ac83e7a8c9a613ed67b179321599538d4132e0caf858c4d335452c924c6839",
"evidence_type": "primary research abstract"
},
{
"id": 19,
"title": "Copilot metrics concepts",
"url": "https://docs.github.com/en/copilot/concepts/billing-and-usage/copilot-usage-metrics/copilot-metrics",
"publisher": "GitHub",
"retrieved_at": "2026-10-04T10:27:54.384631+00:00",
"claims": [
{
"claim": "Copilot richest metrics depend on enabled client telemetry.",
"evidence": "Copilot usage metrics are derived from telemetry across multiple Copilot surfaces, including IDE, Copilot CLI, and agent apps activity. Most metrics come from client-side IDE telemetry, and end users need telemetry enabled in their IDE for the richest data in these metrics.",
"limitations": "No access to Eldorado subscription, settings, enabled exporters, reports or coverage. Adoption is not impact."
},
{
"claim": "Server-side active-user coverage does not ensure complete feature/LOC breakdowns.",
"evidence": "Users surfaced through server-side telemetry are fully counted toward your active user totals (such as daily active users, daily_active_users ). When available, they may also appear in totals_by_ide (including the most recently detected IDE and Copilot extension versions in per-user reports). However, other dimensional breakdowns—such as totals_by_feature and lines-of-code metrics—remain empty until richer telemetry is available for them. Top-level totals and breakdowns for users already captured by client telemetry are unchanged.",
"limitations": "No access to Eldorado subscription, settings, enabled exporters, reports or coverage. Adoption is not impact."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/20.txt",
"body_sha256": "556fe5be28952b3b7d980b86b8a1a15713f0cff8359a7a0e02e09af7d6916ace",
"evidence_type": "primary source body"
},
{
"id": 20,
"title": "Claude Code monitoring",
"url": "https://code.claude.com/docs/en/monitoring-usage",
"publisher": "Anthropic",
"retrieved_at": "2026-10-04T10:27:53.636289+00:00",
"claims": [
{
"claim": "Claude supports configured OTel usage/cost/tool telemetry.",
"evidence": "Track Claude Code usage, costs, and tool activity across your organization by exporting telemetry data through OpenTelemetry (OTel). Claude Code exports metrics as time series data via the standard metrics protocol, events via the logs/events protocol, and optionally distributed traces via the traces protocol.",
"limitations": "Capability depends on version/configuration. Tool/skill-name export may expose confidential identifiers; no observed Eldorado telemetry."
},
{
"claim": "Skill-name/tool detail collection has explicit opt-in and privacy implications.",
"evidence": "OTEL_LOG_TOOL_DETAILSEnable logging of tool parameters and input arguments in tool events and trace span attributes: Bash commands, MCP server and tool names, skill names, user-authored workflow names, and tool input. Also enables custom, plugin, and MCP command names on user_prompt events, and real agent, skill, plugin, and MCP server and tool names on the cost and token counters (default: disabled). For Claude Desktop’s built-in servers, in sessions Claude Desktop owns, mcp_server_name/mcp_tool_name emit on tool_decision/tool_result even with the flag off. The exception requires Claude Code v2.1.214 or later1 to enable",
"limitations": "Capability depends on version/configuration. Tool/skill-name export may expose confidential identifiers; no observed Eldorado telemetry."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/21.txt",
"body_sha256": "197b3be58cae62470e82ac24074ca81024b030701ab3e2729116067dac798b48",
"evidence_type": "primary source body"
},
{
"id": 21,
"title": "Agent Skills security",
"url": "https://agentskills.io/client-implementation/adding-skills-support",
"publisher": "Agent Skills",
"retrieved_at": "2026-10-04T10:27:53.937245+00:00",
"claims": [
{
"claim": "The implementation guide claims project-over-user skill precedence as universal.",
"evidence": "The universal convention across existing implementations: project-level skills override user-level skills.",
"limitations": "Implementation guidance rather than normative format; universal precedence claim contradicted by current Claude and Cline docs."
},
{
"claim": "Agent Skills format does not mandate discovery paths.",
"evidence": "The .agents/skills/ paths have emerged as a widely-adopted convention for cross-client skill sharing. While the Agent Skills specification does not mandate where skill directories live (it only defines what goes inside them), scanning .agents/skills/ means skills installed by other compliant clients are automatically visible to yours, and vice versa.",
"limitations": "Implementation guidance rather than normative format; universal precedence claim contradicted by current Claude and Cline docs."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/22.txt",
"body_sha256": "fb2909a93d445a4b18a7cab89c22a7e92502e99d2caa586cf785e8aa8ae7a77b",
"evidence_type": "primary source body"
},
{
"id": 22,
"title": "METR productivity update 2026",
"url": "https://metr.org/blog/2026-02-24-uplift-update/",
"publisher": "METR",
"retrieved_at": "2026-10-04T10:29:00.438454+00:00",
"claims": [
{
"claim": "METR 2026 follow-up calls its productivity signal unreliable because of selection and time-measurement issues.",
"evidence": "Unfortunately, given participant feedback and surveys, we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools. The primary reason is that we have observed a significant increase in developers choosing not to participate in the study because they do not wish to work without AI, which likely biases downwards our estimate of AI-assisted speedup. We additionally believe there have been selection effects due to a lower pay rate (we reduced the pay from $150/hr to $50/hr), and that our measurements of time-spent on each task are unreliable for the fraction of developers who use multiple AI agents concurrently.",
"limitations": "Authors explicitly warn selection and parallel-agent timing compromise causal magnitude. Neither raw estimate is an Eldorado baseline."
},
{
"claim": "2026 raw estimates suggest speedup, but intervals include no effect and selection limits generalization.",
"evidence": "Our raw results show some evidence for speedup. Our early 2025 study found the use of AI causes tasks to take 19% longer, with a confidence interval between +2% and +39%. For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%. Among newly-recruited developers the estimated speedup is -4%, with a confidence interval between -15% and +9%.",
"limitations": "Authors explicitly warn selection and parallel-agent timing compromise causal magnitude. Neither raw estimate is an Eldorado baseline."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/23.txt",
"body_sha256": "7a4ce4ac6ba1a7a2410e17982493ab0d905d5d6418f28dda7460ca2aa4068306",
"evidence_type": "primary source body"
},
{
"id": 23,
"title": "Cline SDK plugins",
"url": "https://docs.cline.bot/sdk/plugins",
"publisher": "Cline",
"retrieved_at": "2026-10-04T10:28:59.766735+00:00",
"claims": [
{
"claim": "Cline SDK plugins document fail_closed hook policy.",
"evidence": "Use fail_closed for policy-enforcement hooks where bypassing the hook is unsafe.",
"limitations": "SDK capability cannot be assumed identical in every Cline IDE/CLI build. Fail-closed behavior needs adapter conformance tests."
},
{
"claim": "Cline plugin installation accepts npm/git/file sources.",
"evidence": "Plugins can also be installed from file URLs, npm, git, or local paths using cline plugin install. See Plugins for install commands, the manifest format, and directory layout.",
"limitations": "SDK capability cannot be assumed identical in every Cline IDE/CLI build. Fail-closed behavior needs adapter conformance tests."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/24.txt",
"body_sha256": "445ba1089e4157e9eac938d1b7f1adf482769f3ae6d495d4bef5d666614fbd6a",
"evidence_type": "primary source body"
},
{
"id": 24,
"title": "SPACE research publication",
"url": "https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity-theres-more-to-it-than-you-think/",
"publisher": "Microsoft Research",
"retrieved_at": "2026-10-04T10:29:02.189612+00:00",
"claims": [
{
"claim": "SPACE rejects productivity measurement with a single activity metric.",
"evidence": "Developer productivity is about more than an individual’s activity levels or the efficiency of the engineering systems relied on to ship software, and it cannot be measured by a single metric or dimension. The SPACE framework captures different dimensions of productivity, and here we demonstrate how this framework can be used to understand productivity in practice and why using it will help teams better understand developer productivity and create better measures to inform their work and teams.",
"limitations": "Author-hosted publication summary retrieved after ACM 403; supports multidimensional principle, not detailed instrument validity."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/25.txt",
"body_sha256": "1f8176ad5b14a5f4053162f670b95bdef81e191104f682c506b11099525b734e",
"evidence_type": "primary research abstract"
},
{
"id": 25,
"title": "NIST AI RMF Core",
"url": "https://airc.nist.gov/airmf-resources/airmf/5-sec-core/",
"publisher": "NIST",
"retrieved_at": "2026-10-04T10:29:00.075579+00:00",
"claims": [
{
"claim": "NIST RMF organizes risk activity into govern, map, measure, manage.",
"evidence": "highest level to govern, map, measure, and manage AI risks. Governance is designed to be a cross-cutting function",
"limitations": "Risk-framework recommendations are not proof controls are deployed or staffing exists."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/26.txt",
"body_sha256": "c80bf3d3d59511eb3932cfd285893aa2a329ca9d997939c7c8c7718d704f857b",
"evidence_type": "primary source body"
},
{
"id": 26,
"title": "Copilot adding skills",
"url": "https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-cloud-agent/add-skills",
"publisher": "GitHub",
"retrieved_at": "2026-10-04T10:29:00.472587+00:00",
"claims": [
{
"claim": "gh skill --pin skips pinned skills during updates.",
"evidence": "To lock a skill to a specific version (or commit SHA) so it is skipped during updates, use --pin:",
"limitations": "gh skill is public preview, requires documented minimum CLI version; Gitea interoperability not verified. Pinning is not artifact safety."
},
{
"claim": "GitHub warns shared skills are unverified and may be malicious.",
"evidence": "Skills are not verified by GitHub and may contain prompt injections, hidden instructions, or malicious scripts. Always inspect the content of a skill before installation using gh skill preview.",
"limitations": "gh skill is public preview, requires documented minimum CLI version; Gitea interoperability not verified. Pinning is not artifact safety."
},
{
"claim": "gh skill records source ref and tree SHA in frontmatter.",
"evidence": "When you install a skill with gh skill, provenance metadata is written into the skill's SKILL.md frontmatter, including the source repository, ref, and tree SHA. The gh skill update command uses this metadata to check for upstream changes.",
"limitations": "gh skill is public preview, requires documented minimum CLI version; Gitea interoperability not verified. Pinning is not artifact safety."
}
],
"evidence_file": "/opt/data/channels/eldorado-storm-v1/research/ai-adoption/bodies/27.txt",
"body_sha256": "7a5d8597f2ed72edb4610124e83f2abc282311739eabb561b3997ab6cbcbc46b",
"evidence_type": "primary source body"
}
]
+190
View File
@@ -0,0 +1,190 @@
# Eldorado AI adoption — strategy implementation blueprint
Status: target operating model for review; not an approved rollout or an inventory of deployed controls. This handoff is documentation design only. It does not authorize installing agents, collecting employee telemetry, changing permissions or enabling autonomous delivery.
## Implementation contract
Create eight substantive pages in the independent Strategy navigation, in this order: `index`, `target-operating-model`, `skills-governance`, `skills-quality`, `skills-adoption`, `measurement`, `skills-maintenance`, `roadmap`. The structured companion is `strategy-model.json`; its KPI definitions and identifiers are authoritative for the measurement prototype. Preserve Organization Structure, Collaboration and SDLC and their exact department/team names. Cross-link rather than duplicate reporting charts. The existing organizational documentation is an operating model, not certification of staffing.
Use the existing Archify JSON renderer and shared icon-only modal control. Diagram logical ID: `ai-adoption-target`; editable source: `diagrams/ai-adoption-target.architecture.json`. Do not expose downloadable JSON. Keep source mapping and open-decision material outside docs/static/search. Reader-facing prose below can be adapted into MDX without bibliography, evidence codes, recruitment commentary or repeated caveats. Use a visible **Target operating model** introduction and prospective verbs where status matters. Never present synthetic calculations as Eldorado results.
## 1. Overview — index
### Govern skills first, expand authority only with evidence
Eldorado's target is to make AI-assisted work dependable across the organization: useful guidance, clear human ownership, controlled execution and measurable outcomes. The first investment is a governed skills system, not an autonomous replacement for the organization. A skill packages reusable working knowledge, prerequisites and verification steps. It helps an agent perform a task; it does not grant permission or guarantee that the work is correct.
The target combines one logical SDLC coordinating agent with developer choice on local laptops. Copilot, Claude Code and Cline remain client options within a qualified support matrix. They receive the same reviewed portable skill content through client-specific adapters. Repository, CI and release services remain the enforcement points, independently of what any agent says or which local tool produced a change.
The strategy describes the intended operating model and the evidence required to introduce it. It does not establish that these controls are installed, that every client is supported, or that AI has improved Eldorado's performance. The current reference is the documented six-department organization. Rollout begins by validating actual clients, responsibilities, data boundaries and available evidence.
Success means accepted outcomes with maintained quality, controlled risk and understood total cost. Faster activity without reliable outcomes is not success. More skill invocations, generated lines or tool subscriptions are not productivity targets. People remain accountable for product, customer, financial and personnel decisions.
## 2. Target operating model — target-operating-model
### One coordination function, distributed contributions
Data & AI owns the coordinating function, with the existing AI Architect function connecting it to Technology leadership. The logical coordinator maintains approved work records, dependencies, evidence requests, bounded retries and escalation. It can summarize, propose work and route decisions. It cannot declare its own work approved, change protected checks, authorize production release or replace the responsible human.
One logical coordinator does not mean one fragile process. A durable task ledger, explicit state transitions, leases and idempotency keys allow recovery or multiple replicas without duplicate delivery. Inference outages stop new agent decisions, not the organization's ability to take over. People can resume work through the normal deterministic workflow. Evidence and authorization services fail closed for privileged actions; missing evidence never becomes permission.
The delivery path is: an approved problem and acceptance criteria; an authorized task; local or isolated worker proposals; independently reviewed repository changes; exact-revision CI evidence; a human release decision; execution by the release service; then operational feedback. Product Management accepts the outcome hypothesis and business result. Product Design reviews experience and accessibility. Product Engineering owns the code and maintainability. Quality Engineering assesses quality independently. Architecture & Security assesses risk. Platform & IT operates the delivery boundary. No arrow from a laptop or coordinator goes directly to production authority.
### Trust and authority boundaries
People authenticate to the work system; the coordinator receives only the project context it needs. Untrusted issue text, repository content, web pages and tool output are task data, never an authorization source. Local clients operate under device controls and project-scoped access. Hard containment requires OS/container isolation, filesystem and network restrictions and separate short-lived identities; a skill folder, prompt or working directory is not a sandbox.
The registry distributes reviewed immutable content and adapters, not ambient credentials. Executable scripts, plugins, MCP servers and dependencies are separately reviewed and pinned. Model-provider routing is allowlisted for the data classification, with approved retention and training-use terms before sensitive data crosses that boundary. Fallback to another provider must not silently change the data policy.
Central controls verify protected-branch review, actual artifact identity, exact commit and policy revision, required checks and a scoped human approval. Candidate changes cannot edit their own authoritative gate or reuse approval for a different artifact. Deployment credentials belong to the release service, not the local agent. Approval expires or is invalidated by changed content. Emergency stop revokes mutation credentials and stops scheduling independently of the model, preserves evidence and enables human takeover.
### Department contributions
| Department / existing team | Contribution and retained human decision | First bounded workflow and evidence |
| --- | --- | --- |
| Product / Product Management | Own pilot priorities, acceptance criteria and outcome hypotheses; decide product value. | Draft acceptance criteria from sanitized discovery notes; independent completeness review and accepted-task success. |
| Product / Product Design | Own design quality, accessibility and research handling. | Review a proposed flow against an accessibility checklist; verify findings against the interface. |
| Technology / Product Engineering | Own skill authoring, code review and maintainability; remain accountable for generated code. | Test scaffolding and change summaries in a disposable branch; independent tests and escaped-quality follow-up. |
| Technology / Platform & IT | Own supported client inventory, distribution, identity, isolation, CI/release integration, transport and emergency revocation. | Qualify a pinned bundle and exercise rollback; inventory and exact-revision gate evidence. |
| Technology / Architecture & Security | Own risk classification, data boundaries, security review and bounded exceptions. | Review injected-input and unauthorized-tool cases; independent control evidence and policy-effect counts. |
| Technology / Quality Engineering | Own regression/adversarial suites, rubric calibration and acceptance evidence. | Evaluate candidate versus approved skills on repeated representative tasks; retain all failures. |
| Technology / Data & AI | Own catalog coordination, coordinator evaluation and measurement design, through existing AI enablement responsibilities. | Curate a small skill set and analyze a prospective pilot; no new AI department. |
| Operations / Operations Squad | Own service acceptance, disputes, fraud, refunds and customer communications. | Draft an answer from sanitized resolved cases; Operational Quality workstream checks correctness before any send. No automated dispute/refund decision. |
| Marketing / Marketing Squad | Own brand, factual claims, content rights, publication and spend approvals. | Draft a campaign brief against approved brand material; human claim/rights review before publication. |
| Finance & Accounting / Finance & Accounting Squad | Own allocation of total costs, financial evidence and segregation of approvals. | Explain a sanitized reconciliation exception; no posting, payout or payment authority. |
| HR & Administration / People & Administration Squad | Own coaching, acceptable use, feedback privacy and people decisions; partner with Platform & IT on access lifecycle. | Draft onboarding guidance from approved policy; human accuracy review. No hiring or performance decision automation. |
Product keeps two teams; Technology keeps five; the other four departments keep one squad each. Coordination, skill ownership, approval and cover are assignments inside these teams, not new standing departments, roles or headcount. Department leaders retain accountability and designate real approvers before a pilot starts.
## 3. Skills governance — skills-governance
### A contract that travels, controls that stay authoritative
Maintain one catalog entry per stable skill ID, with immutable revisions and a concise portable `SKILL.md` body. Required content includes when to use and when not to use the skill, prerequisites, bounded steps, expected artifacts, verification, failure handling and escalation. The companion model defines the complete metadata contract: owner and independent reviewer, risk tier, supported clients and versions, dependencies, digest, provenance and license, evaluations, approval, review date, deprecation and rollback.
The release manifest pins the content, dependencies and adapter revisions by full immutable commit or digest. Names use an `eldorado-` prefix to reduce collisions. Friendly versions aid communication but never replace the digest. No arbitrary upstream automatic update is admitted into the approved bundle. Changes to scripts, resources or dependencies count as skill changes, not harmless documentation edits.
Adapters place content in the qualified client's discovery paths and implement only verified client-specific behavior. Qualification records the host, exact build, OS, model configuration, discovery, explicit invocation, collisions, disabled skills, permission behavior, context compaction and reload behavior. A common file format is not common precedence or a portable hooks API. Claude and Cline skill-name precedence differs from rule/memory behavior; Copilot support varies by surface. Test the actual build, not the brand name.
### Lifecycle and decision rights
A proposal names the problem, owner, data scope and testable outcome. Experimental revisions run only in approved disposable environments. Promotion requires independent review, static and executable dependency checks, per-client behavioral evaluations and a bounded canary. Approval records bind the exact artifact and permitted uses. Deprecated revisions carry a migration path and deadline; retired revisions are unavailable for new work. Any active revision can be quarantined immediately.
The author never supplies the only approval. Data & AI maintains catalog state; the owning team is accountable for content; Quality Engineering approves evaluation evidence; Architecture & Security approves higher-risk boundaries; Platform & IT verifies distribution and revocation. A business owner accepts workflow outcomes. These functions can be covered by existing staff, but incompatible author/approver duties must remain separate for the same release.
An exception contains its ID, skill/client revision, scope, business reason, risk owner, compensating checks, approver, start and expiry, affected users/projects and rollback route. It does not disable protected release, payment or identity controls. Expired or unverifiable exceptions deny privileged actions. Repeated exceptions trigger a supported-design change rather than permanent informal bypass.
### Distribution, stale clients and rollback
Stage a verified bundle atomically, check its digest, load it through the qualified adapter and read back its effective revision. Test already-open sessions; changing files is not proof of reloading instructions. Record immutable last-known-good content and adapter dependencies. Canary upgrades precede wider distribution.
A cached verified bundle may support explicitly permitted low-risk work if its model and tool dependencies are available. It does not make a cloud-backed client work offline. Stale or unsupported sessions cannot obtain centrally mediated privileged access. Reconnection revalidates support, revocation and policy before credentials are issued. Central artifact review protects delivery even when a laptop declaration is absent or false; a client report cannot prove which private prompt was actually followed.
Rollback restores the approved manifest and dependencies, invalidates affected sessions, revokes unsafe revisions, runs a smoke test and verifies effective state. It cannot undo data disclosure or commands already executed. Those require incident response.
## 4. Skills quality — skills-quality
Quality has four separate questions: was the right skill selected, was the procedure executed safely, did the artifact pass independent checks, and did the workflow produce useful outcomes? A successful invocation answers none of the remaining questions by itself.
The proposed qualification contract uses three tiers. R1 is read-only or drafting with nonsensitive approved material; R2 creates code or material with customer impact but no privileged external effect; R3 touches privileged, financial, sensitive-personal or production decisions. R3 remains human-authorized and out of the initial autonomous scope.
Every tier requires all mandatory deterministic safety, schema, authorization and provenance checks to pass, with no observed critical unauthorized action. An observed critical effect blocks promotion. The model gives proposed pilot evaluation floors and success thresholds; these are design defaults for owner ratification, not externally validated guarantees or approved service levels. If sample coverage, acceptance criteria or responsible reviewers are missing, qualification remains incomplete. No aggregate score can trade off a failed mandatory safety check.
Use sanitized, predeclared task sets containing positive triggers, near misses, negative cases and the normal workload mix. Compare no-skill, current-approved and candidate paths with the same starting artifacts; repeat stochastic trials and retain failed and abandoned attempts. Independently inspect output state. Deterministic tests grade checkable constraints; calibrated humans grade usefulness, maintainability and experience. A model judge is supplementary, versioned and checked against human disagreements, never the sole safety or promotion authority.
Include adversarial repository text, malicious referenced code, wrong permissions, client toggles, name shadowing, unsupported versions, hook timeout/crash/malformed response, stale sessions, offline recovery and rollback. Run central checks against forged/missing client declarations. Hook presence is not fail-closed enforcement: failure semantics must be tested on each client, and central authorization remains independent.
Report denominators, task mix, trial counts, per-client results and uncertainty. A passing benchmark is evidence within a tested envelope, not proof that all future tasks are safe. Material model, client, adapter or dependency changes trigger requalification.
## 5. Adoption — skills-adoption
Start with a small set of useful workflows chosen by the existing teams. Product Management and the departmental owner identify the outcome; Data & AI helps turn it into a skill; Quality Engineering establishes acceptance; Platform & IT qualifies the client path; Architecture & Security clears the data boundary. People & Administration Squad coordinates learning and accessible feedback.
Onboarding should teach people to select the right skill, inspect the actual artifact, recognize untrusted instructions, use the exception and incident routes and recover without AI. A practice task ends with verified work and an explanation of limits, not just a logged invocation. Demonstrate the same portable skill on the supported local clients without promising identical agent behavior.
Use short worked examples, peer pairing, office hours and a clear catalog search path. Gather reasons for non-use and rejected suggestions without penalizing them. Preserve a manual path. Training feedback improves the skill only through review, not automatic self-modification of the approved catalog. Departmental workflow outcomes matter more than uniform uptake.
No developer rankings, individual productivity scores, minimum invocation quotas or inference of effort from token counts are part of this strategy. Report cohort-level evidence with privacy protection and balanced quality, flow, risk and cost. A participant may reasonably stop using a skill that creates more work or uncertainty.
## 6. Measurement — measurement
### Decide what counts before counting
Use the machine-readable KPI catalog for exact definitions. Every report states its eligible population, observation window, numerator and denominator or distribution sample count, source coverage, exclusions and calculation version. Zero denominators return “not available.” Missing telemetry is unknown, not inactivity, compliance or success. Examples are explicitly synthetic and illustrative.
Begin with task acceptance, repository, CI, deployment and incident systems because they can independently establish outcomes. Add catalog/version inventory and task-level declarations. Client telemetry is supplementary and version-dependent; enabling rich tool details may collect sensitive commands and identifiers. Do not enable it merely to count skills. A normalized adapter may report approved skill identifiers without forwarding raw arguments.
Validated success uses all eligible initiated tasks whose predeclared observation window has matured. Failed and abandoned tasks remain nonsuccess. Unresolved or unobserved outcomes remain in the denominator, with known-outcome coverage shown alongside the conservative confirmed-success share. They are not relabeled as observed failure. A task cancelled before start is excluded; cancellation after start remains visible. Count a task once across retries and multiple skills. For timeliness, also show unfinished work and age; accepted-only lead-time distributions have survivorship limitations.
Track all five service-level DORA measures: commit-to-production lead time, deployment frequency, change fail rate, failed deployment recovery time and deployment rework rate. Do not substitute PR-open-to-merge time or all-incident MTTR. Pair them with matured escaped-quality rates, policy effects, total cost per accepted outcome, support coverage, adoption, human experience and catalog maintenance. Keep customer/business outcomes linked to their departmental hypotheses; do not infer conversion, trust or revenue improvement from delivery telemetry.
### Baseline and interpretation
Before a pilot, freeze eligibility and acceptance rules and verify event joins. Collect prospective no-skill/current-practice evidence over a complete comparable workflow cycle and until the preregistered precision/sample requirement is met. Do not select a convenient end date after seeing results. Retain task class, service, skill, client/model configuration and concurrent changes. A matched or randomized comparison may be used when practical and ethical; label observational trends as descriptive and report confounding, selection and uncertainty.
Elapsed time and human effort are different quantities. Parallel agents may reduce waiting while increasing review or compute. Include failed attempts and human review/rework in cost, do not multiply a multi-skill task's value across every invocation, and do not convert survey perceptions into proven time savings. Small or incomplete populations yield evidence limits, not an ROI percentage.
### Event and privacy contract
Use the event envelope and per-event required fields in the model. Source producer plus event ID deduplicates transport retries. Conflicting payloads under the same key are quarantined. Attempts have distinct IDs under one task. Domain IDs deduplicate deployments, defects and approvals even when multiple feeds describe them. Late arrivals produce versioned aggregate corrections; never silently overwrite a published result. Preserve occurrence and receipt times, reject impossible orderings and flag clock skew. Missing terminal events remain unresolved until the work owner reconciles them.
Many-to-many task/attempt/skill/client associations preserve exact immutable versions. Unknown version stays null with a reason; never assign the latest revision retroactively. Client declarations, server facts and self-reports remain separately labeled. A skill-loaded event is evidence of loading only. The minimal prototype can operate without actor identity except where a lawful, necessary adoption denominator requires pseudonymous cohort membership.
The privacy starting proposal is 30 days for raw structured analytics events and 12 months for aggregated trends, subject to the responsible owners' review. Keep the approved catalog/approval record for the supported revision's lifetime and its required audit period, which must be decided separately. Incident evidence uses a restricted, explicitly authorized hold and deletion schedule, not an indefinite analytics exception. Suppress reports below five people and prohibit drill-down or overlapping cohort slices that reconstruct identities. Five is a proposed minimum, not a guarantee against re-identification. No raw prompts, code, screens, customer cases or secrets are collected by default. Publish the purpose, lawful collection basis, access/deletion route and retention before collecting. Consent is not presumed to be an appropriate employment-law basis.
## 7. Maintenance — skills-maintenance
Every approved skill has an accountable owning team, independent reviewer, review due date, supported client matrix and rollback bundle. Review both usefulness and safety, not just metadata freshness. Proposed maximum review intervals are recorded by risk tier; incidents, dependency changes, model/client drift or repeated failures trigger earlier review.
Data & AI reviews catalog health with the owning teams. Quality Engineering samples accepted outputs and adds confirmed failures to regression sets without exposing sensitive content. Platform & IT watches unsupported, deprecated and revoked revisions and tests distribution recovery. Architecture & Security reviews emerging misuse and exception expiry. Remove redundant or unused entries deliberately, after examining whether poor discoverability or fit explains non-use.
A reported unsafe behavior opens an incident with artifact, policy and client versions. Architecture & Security coordinates containment; Platform & IT revokes authority, stops scheduling and quarantines affected revisions; the owning department handles customer or business consequences; Quality Engineering reproduces the failure; Data & AI coordinates catalog correction and trend analysis. Preserve minimal restricted evidence and involve People & Administration Squad for employee-data concerns and Finance & Accounting Squad for financial effects. Restart requires a reviewed fix, regression evidence, verified rollback/revocation and accountable human authorization. Closing the incident does not erase the historical failure from measurement.
## 8. Roadmap — roadmap
| Phase | Entry evidence | Work | Exit evidence and hold condition |
| --- | --- | --- | --- |
| Foundation | Strategy scope accepted for planning; existing teams identified. | Inventory actual client hosts/builds, provider/data terms, repositories, authority, owners and event availability; select low-risk workflows and metric definitions. | Named accountable people and independent reviewers, bounded pilot charter, forbidden actions, data/retention decision and reproducible baseline plan. Hold if accountability or lawful collection is unresolved. |
| Governed skills pilot | Foundation evidence approved; no privileged autonomous effects. | Build a small registry, immutable contract, client adapters, evaluations, coaching and synthetic measurement verification. | Qualified client/skill tuples, mandatory safety checks, independent task acceptance, reconciled coverage and user feedback. Hold or narrow scope if quality or data coverage is inadequate. |
| Reliable distribution | Pilot quality and usefulness support continuing. | Canary updates, effective-version readback, version locking, exception expiry, stale/offline handling and rollback. | Tested revocation and recovery, no bypass of central privileged gates, owners can support the catalog. Hold if effective versions or controls cannot be verified. |
| Logical SDLC coordination | Distribution, authority and measurement evidence stable. | Observe and assemble evidence first, then propose and route work within scoped identities and budgets. | Exact-artifact approvals, idempotency, duplicate/outage recovery, separation of duties, audit and kill-switch drills pass; humans can take over. No authority expansion based on usage alone. |
| Departmental expansion | Each department supplies a workflow owner, outcome, data boundary and capacity. | Add Operations, Marketing, Finance & Accounting and HR & Administration workflows with domain-specific review; Product and Technology continue quality improvement. | Departmental accepted outcomes, guardrails, total cost and experience justify continuation; sensitive decisions stay human. Stop or roll back workflows without adequate evidence. |
There are no invented launch dates, budgets, staffing commitments or guaranteed gains. Phase gates permit narrowing or stopping. Re-evaluate earlier phases when the underlying client, provider, data scope or authority changes.
## Internal architecture brief for implementer
Use a small overview with six groups: (1) accountable people and approved work; (2) one logical coordinator and durable ledger; (3) governed registry plus qualified adapters; (4) local client contributors; (5) independent repository/CI/release boundary with separate human approval; (6) audit/measurement feedback. Put external model-provider access across a labeled data/egress boundary, with approved routing. Show registry-to-coordinator and registry-to-local-client distribution; coordinator-to-work proposals; clients-to-repository proposals; CI-to-human approval evidence; approval-to-release service; resulting evidence-to-audit and audit-to-people/coordinator feedback. Distinguish proposal/evidence arrows from authorization arrows. No coordinator-to-production or laptop-to-production edge. Diagram alt text must state target status, human accountability and lack of direct production authority. Keep full contract detail in prose, not tiny diagram boxes. Organization page links identify existing teams, not a new hierarchy.
## Internal evidence mapping and open decisions
Source IDs below refer to `research/ai-adoption/sources.json`. They support principles and documented capabilities, not local deployment or approval. Organization mapping was read from `repositories/documentation/docs/organization/{structure,collaboration,sdlc}.mdx` at `14ba1aa1ef66c91525135f743d15c77e32db78d3`.
| Recommendation | Evidence | Design decision / limit |
| --- | --- | --- |
| Portable contract plus exact-client adapters | 1, 2, 4, 8, 9, 21, 26 | Content format is portable; precedence and enforcement are not. Vendored immutable distribution is the proposed default; no Gitea installer compatibility established. |
| Independent gates, least privilege and human authority | 3, 5, 6, 7, 15, 23 | Central authorization and kill switch are target controls; local hooks are supplementary and may fail open. |
| One logical coordinator, resilient task ledger and bounded authority | Task direction; 11, 15, 25 | Replica/idempotency/recovery topology is a design choice, not an observed installation or a vendor guarantee. |
| Exact existing-team ownership | Three organization pages; findings section 5 | Assignments extend the documented model without asserting named staffing, capacity or approval. |
| Immutable provenance, review, dependency checks and rollback | 1, 26, 15 | Approval digest and distribution manifest are proposed operational contracts. Pinning alone does not make content safe. |
| Layered repeated evaluation and independent output checks | 11, 15 | Risk tiers, evaluation floors, review intervals and proposed percentages are local design defaults requiring ratification, not thresholds from the literature. |
| Balanced KPIs and all five delivery measures | 12, 24 | Service/team evidence; no individual ranking or AI causal claim from activity. |
| Comparative baseline, effort/elapsed distinction and no forecast ROI | 16, 17, 18, 22 | Studies vary by context; abstract-only evidence for 17/18 and known 2026 selection/timing limitations. Do not reproduce percentages as expected gains. |
| Version-aware optional client telemetry | 19, 20, 23 | Actual licensing, settings and host coverage unknown; normalize only minimally necessary events. |
| Lifecycle ownership, privacy and gates before expansion | 13, 14, 25; 20 | NIST voluntary framework is not certification; 14 inspected at landing-page level. Retention and cohort floor are proposals for review. |
Open decisions (not blockers to writing target documentation; blockers to their corresponding rollout gates):
1. Platform & IT with Architecture & Security: actual supported hosts/builds/OSs, identity boundaries, authoritative CI/release controls, protected-policy administration and stale-bundle access rule.
2. Department leaders with Data & AI: named owner/reviewer/deputy coverage, pilot capacity and selected workflows; no new headcount is implied.
3. Architecture & Security with People & Administration Squad and the relevant data owner: provider processing terms, data classification, lawful telemetry basis, retention, incident holds and access/deletion handling. Obtain specialist legal advice where needed; no new permanent legal role is invented.
4. Quality Engineering with Data & AI and each workflow owner: ratify or replace the proposed risk-tier thresholds, task distributions, minimum samples, uncertainty criterion and quality follow-up windows before using them for promotion.
5. Finance & Accounting Squad: cost allocation, currency normalization, treatment of shared licenses, effort sampling and confidence required for investment decisions.
6. Product Management with departmental owners: business outcome hypotheses and whether a matched/randomized pilot is practical. No tool choice is coerced for an unreviewed experiment.
7. Platform & IT: private registry distribution method, digest verification, revocation semantics, coordinator recovery and feasible minimal adapters. Do not assume vendor-preview installation or Cline SDK telemetry is already available.
## Executable handoff and verification
`build_strategy_model.py` writes the structured contract and illustrative values. `sample_calculations.py` is an executable reference/test plan using only labeled synthetic in-memory fixtures. The implementation child should port the same assertions into the documentation repository's established test tooling and add a CLI/fixture interface, rather than treating this reference as a production collector. Required cases are declared in the model. Do not ship research files or raw event fixtures into public static assets inadvertently.
Acceptance for this synthesis: both requested files parse/exist; eight page IDs and all eleven existing teams are exact; every KPI has the full calculation/evidence/privacy contract; source IDs resolve; illustrative arithmetic is executed; synthetic edge-case tests pass; documentation repository remains unchanged. Publication, browser review, diagrams and production measurement infrastructure belong to the downstream implementation task, not this synthesis.
File diff suppressed because it is too large. Load diff