CASE STUDY: HARNESS ENGINEERING
How I make Claude follow the rules
By Shawn Clybor, October 2026
I run my consulting firm’s day-to-day work through Claude, under written rules for how it handles client records, email and files. This case study shows how I measured whether Claude followed those rules, and the checks I built when it didn’t.
PhD, Northwestern
Doctoral degree in history (2010) and published scholar
Fifteen years teaching
Universities including Northwestern, Utah State and Manhattan College, and secondary schools
IWAI Certified Consultant
Published case study
How I make Claude follow the rules, measured before and after
Short answer
Harness engineering means building the instructions, tools and checks that sit around an AI model. I run my firm’s operations through Claude under written rules. One rule requires a planning step before Claude changes anything, and Claude skipped it in 72 percent of the sessions where it made a change. I replaced the rule with a check that blocks the change until the step has run. In the 30 days after, Claude skipped the step in 2 of 242 sessions. The planning check is published as a free, open-source plugin, think-first, with 21 tests and a script that breaks it 18 ways to prove the tests catch each break.
What problem does harness engineering solve?
I run Clybor Consulting’s day-to-day work through Claude: client records, email drafts, calendars and files. That system runs automated guards that screen out personal information, and when a client firm wants its material kept in its own systems, I work inside the firm’s tenant. Each area has written rules, and most of them were written after something went wrong, such as a reply sent on the wrong email thread.
I could see in individual sessions that Claude sometimes ignored a rule, but not how often.
How did I measure whether Claude followed the rules?
One rule says Claude must work through a plan with a reasoning tool before it changes anything in another system, such as a client record, an email, a calendar or a file. I went through every session from June 17 to July 16, 2026. Claude made a change like that in 322 of them. It ran the planning step in 90 and skipped it in 232, which is 72 percent.
I replaced the rule with a check, then ran the same count with the same script from September 7 to October 6, 2026. Of 242 sessions, Claude ran the planning step in 240.
What can a check do that a written rule can’t?
A check is a short script, a Claude Code hook, that runs before each action Claude takes and can block it. The planning check blocks any change until the planning step has run. A second check blocks Claude from touching email, client records, calendars or files until it has loaded the written rules for that area. The block message names the rules to load, and Claude loads them and retries.
I sort my rules into three groups: rules a script enforces by blocking, rules a script reports on without blocking, and rules that need judgment and stay in writing.
What did I test, and what did it catch?
The checks have their own tests
Each check has its own tests: 21 cases for think-first, 18 for read-first and 10 for guard-resolve. think-first is the public one; I run the other two myself. guard-resolve also has a mutation test that breaks the launcher four ways and fails unless the tests catch every break.
A crashed check let everything through
In August 2026 I found that when one of these checks crashes, Claude carries on as if the check had approved the action. guard-resolve makes a missing or broken check block the action instead.
A check that didn’t fire
In September a scheduled run drafted an email without loading my email rules, and the check didn’t block it. The check kept no record of the actions it allowed, so I couldn’t find out why. It now logs every decision, including the ones where it couldn’t decide.
The decision log
From September 21 to October 6, 2026, the check logged 16,250 decisions. It blocked 111 actions until the right rules were loaded. It let 154 actions through because it couldn’t read the information it needed, and logged each one. I turned it off deliberately 14 times.
What were the results?
- Before the check, Claude skipped the planning step in 232 of 322 sessions (72 percent). After it, 2 of 242 (1 percent).
- think-first is MIT-licensed. The rest of my Claude Code tooling, including the fail-closed launcher for security checks, is public in clybor-claude-tooling.
How does this apply to a law firm or a team using Claude?
The same approach applies to a firm’s AI use policy. Some parts of a policy can be enforced by a script: Claude must load the firm’s rules before it opens a client file, a draft can’t go out until its citations are verified, and every allowed or blocked action is logged. The parts that need judgment stay in the written policy and in staff training.
Who did the work?
I wrote the rules, the checks, the tests and the measurement scripts, using Claude Code. For over a year I also led the technical coaching calls for Innovating with AI and taught 26 lessons, one of them a live build of a system that scores an AI’s answers for faithfulness and relevance. My essay What is Harness Engineering? on Medium covers the background.
Want checks like these around your team’s Claude?
Thirty minutes, no slide deck. Tell me where a Claude mistake would cost you most, and I’ll tell you which of those mistakes a check can catch.
Or email shawn@clyborconsulting.com.