You save your best prompt, the one that got an agent through a clean refactor. Three months later you paste it into another repo and it goes badly: the agent removes a retry you needed and edits a test until it passes. The prompt was the same. What was missing is everything else you knew that day.

The prompt was addressed to the agent, and the rest was addressed to nobody. You handled it yourself, or watched for it, so none of it made it into the saved prompt. That’s why I think a prompt library saves the wrong thing. The instruction was the easy part. What’s worth writing down is what you’d tell a colleague before they start, with the prompt as one line inside it.

The refactor handover

Hand the same refactor to a person and you wouldn’t forward them the prompt. You’d tell them to make sure the tests pass before they touch anything, to get a plan out of the agent and read it before any code exists, to watch for it editing the tests, and that they’re done when the same tests pass without changes. That’s setup, order, pitfalls and a done check. It’s addressed to the person running the agent, not to the agent.

Pitfalls from real runs

Of those four, the pitfalls carry the most. An agent can produce a reasonable plan for a refactor without your help, and it can list the generic risks too. It can’t tell you which ones actually hit when you ran this kind of task: that it changed behavior while restructuring, tidied away error handling you needed, or wandered into cleanup nobody asked for. Those are observations, and they only exist once someone has made the mistake and written it down.

Most of what you’d tell that colleague is this record. Each pitfall is a past failure turned into a check: some become a line in the prompt, the rest something you look for in the diff. It’s the same habit as growing an AGENTS.md, the file of repo instructions an agent reads before it starts, one mistake at a time. But AGENTS.md holds what’s true of one repo, for the agent to read. These notes hold what’s true of one kind of task, in any repo, for the person running it.

One page per kind of task

I’m putting these in a repo called ai-playbooks: plain Markdown, one page per kind of task. The first six are refactoring, legacy redevelopment, clean-room implementation, test backfill, dependency migration and codebase onboarding, and every page has the same six sections:

  • When to use it, and when to pick a different page
  • Setup
  • Steps, with the prompt for each
  • Pitfalls
  • Done check
  • Example, a link to a real run

Most pages follow the same four steps: research, plan, implement, check. The agent reads the code before it writes any. You review the plan, because one wrong line there turns into hundreds of wrong lines of code. Implementation starts in a fresh session that gets only the plan. Then you check the result against something the agent can’t talk its way around, like the original tests and a diff of the test files.

None of that sequence is mine. It comes from published work on context engineering and the research, plan, implement workflow, and the repo has a short principles page that says why each step is there. An agent works from the text it has in view during a session, and it tends to do worse as that text fills up with things it doesn’t need. Most of the steps keep that text small and make the checkpoints cheap.

Stale prompts

The prompts inside the steps are the part I expect to rewrite. A prompt is tuned to the agent you wrote it for, and it goes stale when the next version behaves differently. “Tests must exist before you refactor” doesn’t go stale, and neither does “check the diff on the test files”. A prompt library ages with the agent. Most of each page is about the task, so I expect it to outlast the prompts inside it.

A skill file with extra steps

There’s a fair objection to all this: agents need less hand-holding with every release, and they can already load skill files, instructions packaged for one kind of task. Isn’t this a skill file with extra steps? A skill file could carry the prompts. It can’t carry “read the plan before any code exists” or “check the diff on the test files”, because the agent isn’t the one doing those. Most of each page is for the person running it, and the prompts sit inside it because that’s where they get used. And written guidance doesn’t always help. Every line the agent reads takes room in the same window it needs for the work, and a line that doesn’t apply to this task is just noise. A page of advice is not automatically an improvement.

So pages come from real runs, and they get shorter when a run shows a line wasn’t needed. My first design for the repo went the other way, with evaluation harnesses, approval tiers and a threat model on every page. Each addition was easy to justify, and together they made something I’d never open, so I cut it back to pages short enough to read before starting work. Even short, each page is one more thing to keep current, and it only helps if someone reads it before starting.

Takeaways

  • A saved prompt keeps the instruction and loses what you did around it: the setup, the checkpoints, what you watched for.
  • Write down what you’d tell a colleague before they start, with the prompt as one line of it.
  • An agent can list generic risks. The pitfalls worth writing down are the ones your own runs hit.
  • Expect to rewrite the prompts when the agent changes. The rest should outlast them.
  • Start each page from real runs, and cut any line a run shows you didn’t need.

Most of what made that first refactor go well was never in the prompt.