A skill is a page of instructions that teaches an AI agent to do one job well. Our System Pulse tool, which lets an agent read a Windows machine’s health, is getting a library of them: a health check, a crash investigation, a hardware-error investigation, and more. The trouble with skills is the same as the trouble with any exam. The person who writes the instructions can see the questions, and the instructions improve until they pass.
So before the library’s first skill was committed, we sealed the questions.
What we sealed, and when
The test is a list of 19 cases, each a real situation an agent might be asked about. We fixed the list first, hashed it (SHA-256, starting 85a49a17; the hash fixes the list’s exact bytes), and wrote the hash and a clock-read time, 17:55:47 UTC on 12 October 2026, into a file committed ten seconds later. The times come from our own clock and commits, not an outside timestamp. The first skill of the library was committed at 18:02:09, a little over six minutes after the seal commit. One sealed case’s expected answer changed after sealing, because the machine’s log rolled over; an errata file records it, and the list is unchanged.
Our own seal record says the clock was read before any skill text existed; git can show only the order of commits.
Then came the split. About a quarter of the cases are for development: the five the author may study while writing. The other fourteen are held back for scoring. The five follow from the committed list’s own hash, so anyone can recompute the split; we did, and it matches. That makes the split reproducible. It does not prove nobody tried other versions of the list to get a split they liked.
The part we can’t claim
The author of the skills read every case, sealed ones included. The seal records when the list was committed, not that it was the first version of the list. The protocol says to iterate on development cases only. Nothing enforces that, and it can’t stop knowing the scored cases from shaping the writing. A stronger check is a set of cases a second agent adds from real events the author doesn’t see before scoring. That set doesn’t exist yet. Until that run, every result we report will say which kind of case it comes from: development, author-sealed, or independent.
The harness taught us something first
Before the library’s first skill was committed, we ran a no-skill baseline on the five development cases: five Sonnet-class subagents in Claude Code (exact model version not recorded), one per case, with System Pulse’s reader tools, minus two readings held back. The server’s own instructions were present; “no skill” means no library skill. The first run forbade file reads, to keep the cases from leaking. That broke something real. When a tool’s answer is large, Claude Code saves it to a file and shows the path, and System Pulse’s own instructions tell the agent to read it. Our crash-history reading is large enough to trigger that even for a single result. With file reads forbidden, three of the five agents could not see it at all. Rerun with those saved answers readable, all three reached it, and the author judged each answer much stronger; the first runs were voided as baselines.
We caught it before scoring began. The fix was a rule, not a patch: a test harness keeps what real use allows (here, reading the client’s own saved answers) and restricts only what would leak the cases (every other file).
Reading the whole saved crash answer costs a lot of context, and a good skill might spend less.
What we don’t know yet
Whether any skill beats the baseline. The baseline already read coverage carefully, named filing times as filing times, and refused to give a verdict on a hardware return. A skill has to do better than that, and the scoring has not happened.