How it works
AI listens, code checks, and a person decides. A model reads the transcript and fills the CRM fields. Every item it returns quotes the words it came from. Code confirms each quote is really in the transcript and each number is in the quote. It holds anything that fails. The follow-up email is written only from what passed. A second check reads every number, date, name, and promise in it before anyone sends it.
What's AI, what's code, and what's a person
- CodeThe callsScripted by hand and voiced line by line with a text to speech model
- AI stepTranscriptionWhisper on a laptop for each person's track with word timings and confidence
- CodeWho spoke whenVoice activity measured on each person's track
- AI stepThe CRM fieldsA model reads the transcript and quotes its evidence
- CodeThe evidence checkEach quote is in the transcript and each number is in its quote
- A personUncertain wordsFacts that hold a doubtful number or name go to a person
- CodeThe record and the stagePlain rules from a config file
- AI stepThe follow-up emailA model writes from the checked facts alone
- CodeThe claim checkerEvery number, date, name, and promise against the facts
- A personConcessionsA manager approves anything off the price list
- CodeThe scorecardMeasured from the timings with weights you set
- AI stepThe coaching noteA model writes it from the call and the numbers
- AI step
- Code
- A person
How the calls were made
Every call started as an answer key that lists each fact the call has to hold. The script came second and says those facts the way people talk. People start over, cut each other off, and answer with a short "right" or "yeah." A validator confirmed that every fact's words are actually spoken before anything was voiced. Each line was then voiced by a stock text to speech voice. The buyer's side runs through a simulated phone line with a little noise to bring the transcriber's job closer to a real call.
The deal is three calls between Brinewatch and Rhyolite Foodservice. Brinewatch sells cold-chain monitoring. Rhyolite is a regional food distributor with 38 trucks on the road. The nine scored calls are shorter. Two reps talk with nine different buyers. The models never saw five of those calls until the prompts were frozen.
Synthetic audio is easier than a real call. The voices are clean, nobody mumbles, and the scripts hold every fact on purpose. These scores are an upper bound on what the same pipeline would do on real recordings.
The scoreboard
These scores compare each model's fields with the answer key. An item counts when it's in the right field, cites the moment the fact was said or uses its key words, and carries the right number. Precision asks how many of a model's items match a fact in the key. Recall asks how many of the key's facts some item matches. The test calls are the five the prompts never saw during tuning.
| Model | Calls | F1 | Precision | Recall | Held by the evidence check | Empty fields left empty |
|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | Test (five calls) | 0.81 | 0.81 | 0.81 | 1.1% | 28 of 29 |
| Claude Sonnet 5.5 | The deal (three calls) | 0.81 | 0.78 | 0.85 | 0.0% | 14 of 16 |
| Claude Sonnet 5.5 | Tuning (four calls) | 0.81 | 0.75 | 0.88 | 1.1% | 27 of 30 |
| Claude Haiku 5.5 | Test (five calls) | 0.80 | 0.74 | 0.87 | 0.0% | 27 of 29 |
| Claude Haiku 5.5 | The deal (three calls) | 0.76 | 0.67 | 0.88 | 0.0% | 11 of 16 |
| Claude Haiku 5.5 | Tuning (four calls) | 0.78 | 0.73 | 0.84 | 0.0% | 26 of 30 |
| GPT-6 Luna | Test (five calls) | 0.78 | 0.79 | 0.77 | 1.1% | 28 of 29 |
| GPT-6 Luna | The deal (three calls) | 0.76 | 0.70 | 0.83 | 0.9% | 13 of 16 |
| GPT-6 Luna | Tuning (four calls) | 0.76 | 0.74 | 0.78 | 5.3% | 27 of 30 |
What the routed model missed on the test calls
- RisksThe incumbent's add-on is cheap because they're already on its contract (s05)
- StakeholdersEllen Ford, VP of supply chain, and IT security (s05)
- Decision criteriaA few dollars a box won't work at thousands of boxes a week (s06)
- RisksThe vans aren't where they lose boxes, the porch is (s06)
- CommitmentsFigure out the pricing for sensors in boxes (s06)
- What the seller saidThe budget fits five vans and two walk-ins (s07)
- Close dateInstalled before November 1 if Kim likes it (s07)
- ChampionLuis wants it in before the season and agrees to aim for the date (s07)
- CommitmentsBring Kim to the Monday demo (s07)
- Paper processA formal RFP in November inviting three vendors (s08)
- StakeholdersRaj Mehta, COO, and the quality team (s08)
- Paper processCompany information for vendor registration is due by the end of next week, and there are no demos outside the process (s08)
- Next stepsDerek sends the vendor registration information this week (s08)
- CompetitionThe monitoring screen that came with the refrigeration controls only shows temperatures on an office panel (s09)
- Decision processOwen runs it, but anything over $15,000 goes to the partners' monthly meeting (s09)
- Close dateThe partners' next meeting on October 20 (s09)
The models side by side
The same frozen prompts ran on every model. The cost is what each call cost to process at the providers' list prices. The monthly figure is an estimate at a thousand calls a month.
| Model | F1 on test | Held | Email claims flagged | Cost per call | At 1,000 calls a month | Seconds per call |
|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 0.81 | 0.7% | 1 | $0.138 | $137.80 | 38.4 |
| Claude Haiku 5.5 | 0.80 | 0.0% | 1 | $0.013 | $12.70 | 77.3 |
| GPT-6 Luna | 0.78 | 2.3% | 0 | $0.0039 | $3.86 | 57.2 |
Both Claude models think before they answer. Across the twelve calls Haiku wrote six times as many thinking tokens as Sonnet. That's why it takes longer here.
A flagged claim asks a person to look before the email goes out. Any number the facts don't hold gets flagged. That includes a harmless count a model adds on its own like "two concerns."
The page answers the deal calls with Claude Sonnet 5.5. No model reached an F1 of 0.85 on the tuning calls. This one came closest.
Testing the claim checker
The test set holds 30 bad claims across six kinds and 12 fair follow-up emails. All of them were written before the checker was scored. The checker caught 29 of the planted claims and raised a false alarm in 0 of the fair emails.
| Amounts | 5 of 5 |
| Counts | 5 of 5 |
| Percentages | 4 of 5 |
| Dates | 5 of 5 |
| Names and roles | 5 of 5 |
| Promises | 5 of 5 |
The miss is a real number moved onto the wrong thing. The call says half the routes cross dead zones. A planted sentence says 30% of them do. That 30% is real. It's the competitor's price gap from the same call. The routes fact holds no number and the checker has nothing to compare.
Transcription and timing
Each person is recorded and transcribed on their own track. The word error rate compares each transcript with the words the script actually said once both write numbers the same way. Talk time comes from voice activity on each track. It's checked against the mixer's record of exactly when every line played.
| Call | Word error rate | Seller's side | Buyer on the phone line | Buyer before the phone line | Talk time off by |
|---|---|---|---|---|---|
| deal-1 | 1.2% | 1.1% | 1.4% | 1.4% | 0.1 points |
| deal-2 | 0.9% | 0.0% | 1.7% | 1.7% | 0.1 points |
| deal-3 | 4.9% | 1.8% | 7.4% | 4.0% | 0.1 points |
| s01 | 0.4% | 0.0% | 0.8% | 0.8% | 0.3 points |
| s02 | 1.7% | 2.5% | 0.0% | 1.8% | 1.5 points |
| s03 | 0.3% | 0.0% | 0.5% | 0.5% | 0.1 points |
| s04 | 1.0% | 0.0% | 2.4% | 3.9% | 0.0 points |
| s05 | 0.6% | 0.7% | 0.5% | 0.5% | 0.0 points |
| s06 | 1.6% | 2.0% | 0.9% | 0.9% | 0.1 points |
| s07 | 0.4% | 0.0% | 0.6% | 0.6% | 0.1 points |
| s08 | 2.2% | 3.1% | 1.5% | 2.2% | 0.1 points |
| s09 | 0.3% | 0.0% | 0.5% | 0.5% | 2.4 points |
| All | 1.4% | 1.0% | 1.9% | 1.6% | 2.4 at most |
11 of 12 calls land within two points of the true talk time. The voice activity settings were chosen on the four scored calls used for tuning and then checked on the other eight.
Testing the question count
Every question the reps ask in the twelve scripts was labeled open or closed by hand. The rule that sorts them agrees with those labels on 97% of 73 questions. The first version of the rule agreed on 78%. The revision came after reading every disagreement. The newer figure is optimistic for that reason.
The recorded runs
The AI steps ran on my machine and were recorded. This page replays those recordings without calling any service. What you see is exactly what each model returned on the day it ran.
| Model | Step | Calls | Recorded |
|---|---|---|---|
| Claude Sonnet 5.5 | CRM fields | 12 | November 7 |
| Claude Sonnet 5.5 | Follow-up email | 12 | November 7 |
| Claude Sonnet 5.5 | Email after approval | 1 | November 7 |
| Claude Sonnet 5.5 | Coaching note | 12 | November 7 |
| Claude Haiku 5.5 | CRM fields | 12 | November 7 |
| Claude Haiku 5.5 | Follow-up email | 12 | November 7 |
| Claude Haiku 5.5 | Email after approval | 1 | November 7 |
| Claude Haiku 5.5 | Coaching note | 12 | November 7 |
| GPT-6 Luna | CRM fields | 12 | November 7 |
| GPT-6 Luna | Follow-up email | 12 | November 7 |
| GPT-6 Luna | Email after approval | 1 | November 7 |
| GPT-6 Luna | Coaching note | 12 | November 7 |
Privacy
Transcription runs on a laptop and the audio never leaves it. The text steps ran on cloud models through my own accounts. This page itself makes no network request beyond its own files.
What it can't do
The calls are scripted and voiced by text to speech. They're cleaner than real ones. The claim checker reads numbers, dates, names, roles, and promises. It doesn't judge tone or paraphrase. The coaching measures count what can be timed or counted. A good call can still score low on a number. This is a demonstration of a method and not a product. Nothing here was tested on real calls.
The code
These are the project's real files as the build copied them into the page. The featured ones are the checks the page relies on most.