skill-manager/references/description-optimization.md

Description Optimization — Trigger Eval Loop

Source: https://agentskills.io/skill-creation/optimizing-descriptions

The description field is the sole activation trigger. Agents read only name + description at startup. An under-specified description means the skill won't trigger when it should; an over-broad description means it triggers when it shouldn't.

---

Step 1 — Design trigger eval queries

Create ~20 queries: 8–10 should-trigger, 8–10 should-not-trigger.

Store them in evals/eval_queries.json (see assets/evals-template.json for the full format).

Should-trigger queries

Vary along several axes:

• Phrasing: formal, casual, with typos
• Explicitness: some name the domain directly; others describe the need without naming it
• Detail: terse prompts alongside context-heavy ones with file paths and column names
• Complexity: single-step tasks alongside multi-step workflows

The most useful should-trigger queries are ones where the skill would help but the connection isn't obvious — these are where description wording makes the difference.

Should-not-trigger queries (near-misses)

The most valuable negatives share keywords or concepts with your skill but need something different.

Weak negatives (obviously irrelevant — tests nothing):

• "Write a fibonacci function"
• "What's the weather today?"

Strong negatives (near-misses — tests precision):

• "I need to update formulas in my Excel budget spreadsheet" — shares "spreadsheet" but needs Excel editing, not CSV analysis
• "can you write a python script that reads a csv and uploads each row to postgres" — involves CSV, but the task is database ETL, not analysis

Tips for realism

Include in your queries:

• File paths (~/Downloads/report_final_v2.xlsx)
• Personal context ("my manager asked me to…")
• Specific details (column names, company names, data values)
• Casual language, abbreviations, occasional typos

---

Step 2 — Test trigger rates

Run each query through the agent with the skill installed. Observe whether the agent loads the skill's SKILL.md.

A query passes if:

• should_trigger: true → skill was invoked
• should_trigger: false → skill was not invoked

Multiple runs (nondeterminism)

Run each query 3 times and compute a trigger rate (fraction of runs where skill was invoked).

• Should-trigger passes if trigger rate ≥ 0.5
• Should-not-trigger passes if trigger rate < 0.5

Example shell script for Claude Code:

#!/bin/bash
QUERIES_FILE="${1:?Usage: $0 <queries.json>}"
SKILL_NAME="my-skill"
RUNS=3

check_triggered() {
  local query="$1"
  claude -p "$query" --output-format json 2>/dev/null \
    | jq -e --arg skill "$SKILL_NAME" \
      'any(.messages[].content[]; .type == "tool_use" and .name == "Skill" and .input.skill == $skill)' \
      > /dev/null 2>&1
}

count=$(jq length "$QUERIES_FILE")
for i in $(seq 0 $((count - 1))); do
  query=$(jq -r ".[$i].query" "$QUERIES_FILE")
  should_trigger=$(jq -r ".[$i].should_trigger" "$QUERIES_FILE")
  triggers=0

  for run in $(seq 1 $RUNS); do
    check_triggered "$query" && triggers=$((triggers + 1))
  done

  jq -n \
    --arg query "$query" \
    --argjson should_trigger "$should_trigger" \
    --argjson triggers "$triggers" \
    --argjson runs "$RUNS" \
    '{query: $query, should_trigger: $should_trigger, triggers: $triggers, runs: $runs, trigger_rate: ($triggers / $runs)}'
done | jq -s '.'

---

Step 3 — Train/validation split

Split your ~20 queries:

• Train set (~60%, ~12 queries): guide improvements
• Validation set (~40%, ~8 queries): check whether improvements generalize

Keep both sets proportionally mixed (should-trigger and should-not-trigger). Fix the split across iterations.

---

Step 4 — The optimization loop

1. Evaluate on train + validation sets
2. Identify failures in train set only:
   • Should-trigger failures → description too narrow → broaden scope, add more "when to use" context
   • Should-not-trigger false-positives → description too broad → add specificity, clarify what the skill does not do
3. Revise the description:
   • Address the general category that failed queries represent — don't add specific keywords from failed queries (overfitting)
   • If stuck after several iterations, try a structurally different approach rather than incremental tweaks
   • Check that description stays under 1024 characters
4. Repeat steps 1–3 until train set passes or improvement plateaus
5. Select the best iteration by validation pass rate — the best may be an earlier iteration, not the last

Five iterations is usually enough.

---

Step 5 — Apply the result

1. Update the description field in SKILL.md frontmatter
2. Verify it is under 1024 characters
3. Try 5–10 fresh queries (never part of optimization) as a final sanity check

Before and after example:

# Before
description: Process CSV files.

# After
description: >
  Analyze CSV and tabular data files — compute summary statistics,
  add derived columns, generate charts, and clean messy data. Use this
  skill when the user has a CSV, TSV, or Excel file and wants to
  explore, transform, or visualize the data, even if they don't
  explicitly mention "CSV" or "analysis."

The improved description is more specific about what the skill does (stats, derived columns, charts, cleaning) and broader about when it applies (CSV, TSV, Excel; even without explicit keywords).