What counts as a right answer?
A test can only check what you wrote down. Decide what a right answer must say, and what it must never say, before anyone builds a grader.
Pick one where a wrong answer costs money or trust: a price, a deadline, who approves what.
A guess is fine. You need it to price a month.
What does a word check catch, and what does a cheaper model save?
Guess before you build anything. The Check shows you.
Show the answer
Right, because it says 14, and that's a wrong answer getting through. A word check knows only the words you gave it. Put 30 in that question's must_not_say column and the answer fails.
Anthropic's testing guide still favors grading by machine wherever you can, for example by checking for set words, because many questions marked automatically beat a few marked by hand.
Show the answer
About 20 times less. Claude charges per token, a small piece of text, with one price for what goes in and another for what comes out. On Anthropic's pricing page on 10 November 2026, Sonnet 5.5 cost $2 per million tokens in and $10 per million out. Haiku 5.5 cost $0.10 and $0.50, a twentieth on both sides.
That only helps if Haiku still gets your 20 questions right, and the grader is how you find out.
Decide what right means, then let Claude build the grader
Five steps, about 15 minutes. Tick each one when it's done.
-
Put the kit in a folder and open it in the Code tab.
Unzip the kit and open the tidewater-books folder in the Code tab with the mode set to Manual, as in the lessons before.
The Tidewater Books kit
Tidewater Books is a made-up bookshop with an agent that answers customer questions. The ZIP holds shop-rules.txt (eight rules), questions.csv (20 test questions), answers.csv (the agent's answers from last week, with the tokens each one used), prices.csv (three models' prices) and a short README.txt. Unzipping it gives you a folder called tidewater-books.
Download the ZIP (about 3 KB) -
Read what "right" means, and check the prices.
Open questions.csv; it opens in a spreadsheet app. Each question has a must_say and a must_not_say column. For question 10, shipping on a 30 euro order, a right answer must say 4.50 and must not say free. Those two columns are what the grader will check, so this is where "right" gets decided. Two of last week's answers are wrong on purpose. Don't tell Claude which.
Prices change. Open Anthropic's pricing page and compare it with prices.csv, which was read on 10 November 2026. If a price differs, ask Claude to update that line of prices.csv before you go on.
-
Give Claude the build prompt.
Paste it into the Code tab and press Enter.
Build a grader for our bookshop's question agent, so we can test it again every week. The shop's rules are in shop-rules.txt, the 20 test questions in questions.csv, and last week's answers in answers.csv. An answer is right only if it contains every word in its must_say column and none in its must_not_say column. Ignore capital letters. I have no API key, so don't ask Claude anything new and use nothing that costs money: grade the answers already in answers.csv. Never change questions.csv or answers.csv. Then price a month for each model in prices.csv: the average tokens per answer in answers.csv, 50 questions a day, 30 days. Use only the prices in that file, and show the date they were read. Write the score, which questions failed, and a table of cost per question and per month to report.txt. Run it and show me report.txt before you say it's done.
It opens with the job and the data.
"An answer is right only if" turns the two columns into one rule, and the grader marks by that rule alone.
"I have no API key" keeps it free, because last week's answers stand in for the agent. "Never change" keeps anyone from editing the questions or answers to make the score look better.
"Use only the prices in that file" stops Claude from using prices it remembers, which may be out of date.
-
Approve what Claude asks.
Expect a request to create a file in your folder (the grader), then a request to run it, which writes report.txt. Nothing should need installing, and nothing goes over the internet.
Read each request before you approve it. You can say no to any of them and ask Claude why it needs it. Before it says it's done, Claude shows you the report.
-
Ask Claude to explain what it built.
Send this in the same session and read the answer:
Explain what you built as if I've never seen code. One line per file. Then give me one wrong answer your grader would still mark right.
The second part is the one to read closely. A word check passes any answer with the right words in it, so Claude should come back with one like the answer in Predict. Each one it finds tells you a word to add to a must_not_say column before next week.
If you see this instead
- Claude says it can't find answers.csv
- The Code tab is open on a different folder. Open the tidewater-books folder and paste the prompt again.
- The score isn't 18 of 20
- Paste: "Show me each answer and why it passed or failed." Compare the ones it lists with questions.csv. If the rule was read wrong, say which question and ask Claude to fix the grader and run it again.
- The table's prices aren't the ones in prices.csv
- Paste: "Use only the prices in prices.csv, and show the date they were read. Run it again."
- Claude offers to ask Claude the 20 questions for real
- Say no. That needs an API key and costs money. Your prompt asked it to grade last week's answers.
Does your grader catch the wrong answers?
Open report.txt from your folder yourself.
The report is what the grader wrote. Claude's reply in the chat is a summary of it.
You pass when report.txt says 18 of 20 right, with questions 5 and 10 failed, and the monthly prices are $0.45 for Claude Haiku 5.5, $9.00 for Claude Sonnet 5.5 and $18.00 for Claude Opus 5.5, with the date the prices were read, 2026-11-10.
Work one row out yourself. Last week's answers used 2,000 tokens in and 200 out on average. On Sonnet 5.5 that's 2,000 at $2 per million ($0.004) plus 200 at $10 per million ($0.002), so $0.006 a question. Fifty a day for 30 days is 1,500 questions, which makes $9.00 a month.
If you changed a price in step 2, your table will differ from these. Work that row out the same way.
Two questions from memory
Why decide what a right answer must say before you build the grader?
The grader can only check what you wrote down. Rules written after you've seen the answers drift toward those answers, and then a pass tells you nothing.
What do you need to price an agent for a month?
The tokens in and out per question, measured from a real run (here, last week's), how many questions a month, and each model's price per million tokens from Anthropic's pricing page, checked on the day you use it.
Course 4 opens by asking a question from lesson 5.
To test this week's agent, you'd send the 20 questions to it for real. That needs an API key from the Claude Console and costs money: at last week's token counts, 20 questions on Sonnet 5.5 come to about 12 cents. The grader then marks the new answers the same way, for free. The pricing page also lists a 50 percent discount for batch jobs and cheaper repeated input, which course 4 covers.
Course 3 so far
What you've built
- Lesson 1: Gym connector
- Lesson 2: Two agents, one inbox
- Lesson 3: Agent in team chat
- Lesson 4: Stock agent
- Lesson 5: Payment check
- Lesson 6: Grader and budget