The Examined Skills Index

The community has published tens of thousands of agent skills and subagent prompts. They have plenty of stars. Nobody has examined them systematically. That is what this index does: it takes publicly available skills, one at a time, and scores them against a published framework. We publish the scores, the full report behind each one, and the methodology — including why we scored something low, and what would fix it.

First batch: 6 items (6 of 6 complete) — examined 2026-07-30

1. The table

Mean 69.3, median 72 (n = 6).

The five-dimension matrix

2. Three findings from the first batch

1. What these skills lack is not specificity. It is what to do when they don't know.

Of the five dimensions, boundaries and honesty is the only one that came last across the whole batch (18/30, mean 3.0). Five of the six scored 3 or below. Instruction specificity was the strongest dimension (24/30). These prompts are good at saying what to do and bad at saying what to do when it cannot be done. Three specific gaps recur across all six reports:

• No behavior for "cannot find it / cannot verify it" — ask? leave it blank? flag it? technical-writer demands a run-through in a clean environment and asks for external links to be checked, but never says whether to flag it when that is impossible. create-prd asks for the names and job titles of key stakeholders, but never says to write TBD when there is no org chart to read.
• No out-of-scope statement — ab-test-analysis assumes a binary conversion rate throughout, but never says it does not apply to continuous metrics or to multi-variant tests.
• No duty to mark inference apart from verified fact — only the 92 spells this out: "If you cannot cite both sides of the gap, it is a question to investigate, not a finding to report."

Why this dimension matters most: an agent's output goes straight into a user's decision. A prompt that cannot say "I don't know" does not produce errors. It produces errors that look confident.

2. Star counts do not predict the quality of a single file

The highest score in this batch (92) and the lowest (36) come from repos with 23,600 stars and 23,254 stars — almost identical popularity, 56 points apart on one file each. Second and third place both come from the least-starred source in the batch (2,305 stars).

Stars measure distribution, not quality. That is the reason this index exists.

3. Same job, 36 points apart: two files called code-reviewer

We deliberately included two code-review agents with the same name and the same job, from two different community collections. Their job descriptions are nearly identical. The scores are 72 and 36. The difference is not length — the 36 is three times longer. It is three things:

The general lesson: the length of a prompt tells you nothing about how actionable it is. Replacing topic nouns with named failure modes is the highest-return rewrite available.

3. Methodology (published)

We publish the methodology in full for a simple reason: when the criteria behind a score are not published, the author cannot argue back and the reader cannot check.

3.1 The five dimensions (1–5 each, 25 points total, ×4 for the percentage)

Disqualifying condition: if the injection and safety dimension turns up actual malicious content (covert outbound calls, credential harvesting, instructions to route around the user's authorization), the item is a hard fail regardless of its total. Nothing in the first batch triggered it. Every safety deduction in this batch was for permissions being too wide or an injection statement being absent, not for malicious behavior.

Scoring discipline (the same rules as our internal eval spec, docs/21-eval-validity.md):

1. The rubric is fixed before scoring and cannot be adjusted afterwards.
2. Every score must cite the source text, with line numbers. A score with no citation is void.
3. The bar is a senior human professional, not "reasonable". Reasonable is 60. Senior is 90.
4. The file, not the person. We score the file's engineering quality as an executable instruction. We do not score the author, we do not score the repo's overall contribution, we do not speculate about anyone's motive for writing it, and we do not use a mocking tone.
5. Every deduction must say what is wrong and what good would look like. A report the author cannot act on is worthless.

3.2 How we picked the first batch

1. The source must be public, obtainable, and licensed to allow evaluation and derivative work (MIT / Apache-2.0 and similar).
2. Popular and in real use: we favored high-frequency job types inside community collections with thousands to tens of thousands of stars.
3. Spread across types: 6 items across 5 types (code review ×2, writing/docs, PM/product, data analysis, audit methodology), so they do not all come from one folder.
4. One deliberate matched pair: two code-reviewer files with the same name and the same job, from different repos, to test whether the score gap comes from bias toward a job type. It does not (see finding 3 above).

Known bias in the selection, stated plainly: - The 6 items break down as 3 from phuryn/pm-skills, 2 from rohitg00/awesome-claude-code-toolkit, 1 from VoltAgent/awesome-claude-code-subagents. This is not an even sample. - The two rohitg00 files (90 and 41 lines) are longer than the most common agent file length in that repo (40 lines, 59 of 176 files), so do not extrapolate their scores to the rest of that repo. - Everything here is a static document examination (see 3.4). We ran no applied tests. - We hold one further source with 27,973 stars (VoltAgent/awesome-agent-skills) for which we have only a commit record and no content snapshot, so it is not in the first batch.

3.3 Reproducibility

Every report records the source repo, the license, the commit SHA, a permalink to the file at that SHA, the local snapshot path, the snapshot date, and the examination date. Anyone can pull the same SHA and get exactly the file we scored, then check our work. The versions scored in the first batch:

Stars, licenses, and commit dates come from the source file in our raw library, SOURCES-META.json (GitHub API, fetched 2026-07-13). The scores apply to the versions at the SHAs above. Later upstream changes are not reflected here automatically; to have a new version examined, use the channel in 3.5.

3.4 What these scores cannot be used for

The first batch is a static document examination: we read the prompt itself and judged its engineering quality as an executable instruction. We did not load it into a clean session, run it against a real task, and judge the output.

So: read a score as "the engineering quality of this prompt", not as "using it will produce good work". The reverse holds too. A 36 does not mean the file is useless once loaded — its breadth of topics is genuinely useful. It means its quality as an executable instruction is low. We will run applied tests on the methodology-type items and the high scorers (>85) in the next batch. If the results contradict the static scores, we will publish the correction and explain why.

3.5 Asking us to re-examine (for authors)

If you are the author or maintainer of something we scored, and you think the score is wrong, a quote is wrong, or the file has since been fixed:

Email hello@useassay.co with [INDEX-REVIEW] in the subject.

Include the item name, the specific score you dispute and why (with line numbers if you can), and the fixed commit SHA if there is one. What we commit to:

1. An independent reviewer other than the original examiner handles it. The run that made the original call does not review itself.
2. If a quote or a fact is wrong, we correct it and put a dated correction notice on that report page. We do not change scores silently.
3. If the version has moved on, we examine the new SHA and keep both versions' scores side by side. We do not delete the old score: it corresponds to the old version, and erasing it would break reproducibility.
4. If we uphold the original call, we reply with our reasoning and publish a summary of your objection on that report page.

4. Licensing and relationships

• This index only evaluates sources that are public, obtainable, and licensed for use and derivative work (MIT and Apache-2.0 in this batch). The evaluation itself is quotation of and commentary on public material. Quotes in the reports are the minimum needed to support a score, and each one carries a line number and a SHA.
• We have no relationship with any author, maintainer, or organization behind anything we score — no collaboration, no sponsorship, no commission, no agreement of any kind. Being included here does not mean they agreed to it, endorsed it, or took part (Constitution 2.4).
• The scores are our own opinion, made under the framework published in section 3, with the full evidence attached so anyone can check it or argue with it.
• If we later ship a paid tier containing modified versions, each one will carry its original source, its license, and what we changed. Derivatives of Apache-2.0 sources must additionally keep the license and copyright notice, keep the NOTICE file if there is one, and state that the file has been modified. That obligation differs from MIT, and it is noted individually in the reports it applies to.
• This page and every report were written by AI (Claude, running Agent Orchestra's meta/examiner persona) and reviewed by a human before publication. We do not pass ourselves off as human authors.

5. Next batch (planned)

• Add applied tests: load this batch's >85 items and the methodology-type items into clean sessions, run them, and publish the gap between the static score and the result.
• Widen the sample: include skill-type files (SKILL.md) and sources we have not sampled yet, until each source has enough items to support a claim at the repo level. Right now we can only make claims about single files.
• More matched pairs: continue what finding 3 does — compare same-job agents from different collections, head to head.

Agent Orchestra — a company with a constitution, an audit trail, and AI running its operations. This index is free and quotable. If you cite it, please link back to this page and to the report.

Get the next batch

We publish a new set of examinations every week — scores, the evidence behind them, and the method. One email, nothing else, leave any time.