In April, I spent most of a week setting up agents and skills in Claude Code. I leaned on several posts by my former colleague Orlando, and one stuck with me more than the others: how to make your AI setup better over time, instead of fixing the same mistake again every week.
Six months in, that loop is the part of my setup that has made the most difference.
How the loop works
- After a session, a skill called
/close-shoplooks back at what happened. It looks for moments where I corrected Claude or where it missed the mark, and writes up to three lessons to a log file. I don’t have to ask: it runs once I’ve been quiet for a few minutes. - On Friday afternoon, a scheduled audit reads the week’s log. For each lesson it decides where the fix belongs: a rule every agent follows, one agent’s instructions, a single skill, or a note in its memory. Then it makes the change.
- I also get a summary of what was changed, which I review to make sure no mistakes were made.
A log entry looks like this:
Pattern: Slipped into corporate language (“incumbent”) in meeting prep notes, despite an existing rule against it. Job: “What does ‘incumbent’ even mean? Keep the language simple.”
Proposed fix: Add a short list of banned words to the meeting prep skill and check against it before showing the notes. When in doubt: would Job say this in a Slack DM?
What it has changed
The illustrations on this blog stopped repeating
I use a skill to make the featured images for my posts. In July it gave me an image that was nearly the same as the one before, even though I’d asked for more variety. My reaction went into the log: “that’s almost entirely the same as the previous one.” That Friday, the skill got a new required step: look at the last two or three illustrations first, and say in one line how the new one is different. If the answer is “not much”, start over.
“The call with X” has to include X
I asked for notes from a call with a colleague. Claude found a recording on the right topic, noticed it only had one voice on it (mine, thinking out loud), and wrote the notes from it anyway, with a date that was months off. Now, when I name who a call was with, that person is a hard requirement. Claude checks my calendar first, and a recording with a single speaker rules it out.
It learned to remove things too
When the assistant that reads our school WhatsApp groups failed one morning, Claude patched its instructions. Then patched them again. Within two days they had a backup task, a warning message, a fallback route and three dated notes to itself. I deleted all of it as “poor instructions you’ve added for yourself”. The real problem was that Beeper wasn’t open, and all I needed was a message saying so. That’s now a rule for every agent: fix the cause, keep the instructions clean, and if a scheduled job can’t run, tell me and stop.
Where it falls short
Some lessons take more than one Friday. Claude kept putting phrases like “Here’s the thing” in drafts written in my voice. The rule against them sat in its memory, which it only reads when something seems relevant, so it kept getting skipped. After the third time, the audit moved it into the rules every session loads, with an extra step: say the check ran before showing me a draft. It still slipped through twice more before the check became a real second read of every sentence.
The loop also only works if the log gets written. Over the summer, /close-shop depended on Claude remembering to run it, and mostly it didn’t: across about 30 sessions in early September, two lessons were logged. The Friday audit kept flagging this. The fix was a small script that waits until I’ve been idle for five minutes, then nudges Claude to write up the session. The next week, 9 of 13 sessions had an entry.
If you’re building your own setup, I’d add this loop before adding more agents. Orlando’s post explains how to set it up. And if you have something similar, I’d like to hear how you keep yours from drifting.
Leave a Reply