A Writing System That Learns What I Would Have Cut
75% judge-to-human agreement, below the 80% trust threshold
Built with
Not public
The corpus is my own unpublished writing, including client correspondence, so it stays private. The calibration log and the rubric are the parts that carry the argument, and both are described here.
Asking a model to "write like me" produces something that reads like a model imitating a person. The failure is consistent: it over-crafts. It reaches for the constructed punchline and the three-beat close. I do not write like that, and neither does anyone who is just trying to say a thing.
This is the system that fixes it, and it is retrieval rather than fine-tuning, which is the right call for one person's writing on a frontier model. There are five register cards (raw personal, reflective, professional, academic, public) and a corpus of 12 real samples I actually wrote. On a generation it pulls two to five exemplars filtered by register first and topic second, because matching on topic measurably hurts style fidelity: you get a draft about the right subject in the wrong voice.
The part I would defend is the gate. Output goes to a separate judge agent scored against a fixed rubric before I see it, and the judge's agreement with my own rankings is logged every time. That number is currently 75%, across three rounds, against a trust threshold of 80%. It is not trusted, and it does not gate on its own yet.
What those three rounds actually found is more useful than the score. Round one, the judge over-rewarded an aphoristic line I found poster-y. Round two, it picked a punchy construction that the spec and the corpus both endorsed and I rejected anyway, which is how "no constructed punchlines" became an explicit rule rather than a preference. Round three, I rejected both options in two of four pairs, which meant the bottleneck was never the judge. It was the generator over-crafting before the judge ever saw anything.
So the fix went into the generator, not the scorer. That is the whole result: two rounds of tuning the wrong component, and the diagnostic that caught it was logging disagreement rather than assuming the judge was right.
What this does not show
- The judge is below its own trust threshold, so nothing is gated automatically.
- A corpus of 12 samples, all mine. Nothing here generalises to another person's voice without rebuilding the corpus.
- Quality is judged by me and by a model, never by a blind reader. There is no external evaluation.
- The raw-personal register is marked provisional in the system's own notes and is the weakest.