Scientific Citation Retrieval

A course challenge run on Codabench, worked on by a team of four: given a query paper, retrieve the 100 papers it is most likely to cite from a corpus of 20,000, scored on NDCG@10. Around 7.4 relevant documents per query, across 19 domains.

No single retriever wins everywhere, so we built an ensemble of eight. Dense models (SPECTER2, SciNCL, MiniLM) capture meaning; sparse methods (BM25 over title, abstract, full text and per-section, plus TF-IDF) capture exact terms and rare entities; a citation-context pass reads how papers actually reference each other. The eight rankings merge with weighted reciprocal rank fusion at k=6 rather than the conventional 60, with the weights tuned by coordinate descent against the public relevance judgements.

The numbers, in order: a MiniLM dense baseline scored 0.5073. The best single retriever, BM25 over full text, reached 0.5429, beating every dense model we tried, including SPECTER2 at 0.4164. Fusing five signals took it to 0.5760, seven to 0.5918, and eight with retuned weights to 0.6151.

The largest single gain came from a corpus observation rather than a model. Only 18 of 736 citation pairs cross a domain boundary, so 97.6% of citations are same-domain; boosting on that added 0.1026, which is 48.8% of the total improvement. The final held-out score was 0.6965, second on the course leaderboard, where the previous leader sat at 0.6725. Local scores ran a little higher at 0.71 to 0.72, and the gap is honest overfitting: the public judgements were also the tuning set.

Most of what we tried made it worse, which is the more useful half of the result. Cross-encoder reranking dropped the score to 0.4762, and fine-tuning that cross-encoder on our own judgements landed at exactly the same place. An LLM listwise reranker managed 0.5573, at about eighteen cents per hundred queries. Graph expansion over a k-NN neighbourhood changed the score by nothing at all: 0.5760 before, 0.5760 after.

The most instructive failure was a publication-year filter, on the reasoning that a paper cannot cite the future. It fell to 0.3782. The premise was wrong. 68.5% of the papers marked as cited carry a later publication year than the query, with a mean gap of +4.3 years, so the labels track a citation relationship that does not respect chronology. One afternoon of assuming otherwise cost more than any model change gained.

The ceiling is worth stating too. Of 736 relevant documents the pipeline surfaces 663 in the top 100, or 90.1%, and 73 are never retrieved by any of the eight signals at any depth we searched. No amount of reranking reaches those.