Does Retrieval Make Generated Copy Sound Like the Client?

At Chiron, a marketing consultancy, I built the system the content team uses to draft posts in each client's voice. It finds a client's closest past posts and hands them to Claude as examples. The live version is a Claude Code skill that searches the client's published posts for a few keywords, then has Claude draft and check its own work.

My M1 internship was the offline study behind that design, on 1,091 published items from eight clients, with 20 captions per client held out as a 160-item test set. I compared BM25, dense search (bge-small with FAISS) and a hybrid of the two. Dense scored highest on relevance (0.765, against 0.754 hybrid and 0.730 BM25, with 95% bootstrap intervals), but relevance was measured in the same embedding space dense search ranks by, so that comparison flatters it. BM25 gave the most varied examples for a small relevance cost, and it keeps everything in files the team can read. That's why production runs on keywords.

Over 24 briefs and eight client accounts, every retrieval variant raised brand-voice similarity over a no-retrieval baseline, from 0.700 to between 0.737 and 0.759, and all five held after Holm correction (p < 0.001).

Then I checked that score against people. 48 generated drafts were rated blind at the agency in one consensus session, and the score didn't track the ratings (Spearman −0.04 (n = 48)). The report concludes the automatic scores shouldn't be relied on alone. I tested the LLM judge too. Reversing the order of each pair showed no position bias I could detect, and its agreement with a second, embedding-based judge was only fair (Cohen's kappa 0.39).

Two of my own mistakes are in the report. When I scored the live system's drafts against the clients' published posts, the published posts were sitting in the pool they were scored against, so each one matched itself. Excluding those self-matches brought their score from about 0.81 down to about 0.77, a gap about the size of the retrieval effect. And a finding in my June draft, that retrieval made house-style errors worse, didn't survive Holm correction when I re-ran it, so the final version withdraws it.

The content team has used the system every day since the end of July 2026.