§1 The gap between existence and support

As LLMs accelerate paper production, citation hygiene has become an open problem. Tools like HalluCiteChecker and other hallucination-checkers verify whether the cited paper exists. That solves the most egregious failure mode — pure fabrication — but leaves a larger class of citation errors untouched.

A citation can be a bad citation even when it points to a real, well-known paper. The cited work might be weakly related to the claim, tangentially connected, or worst, the claim might mischaracterize what the cited paper actually establishes. These are not malicious; they often come from sloppy literature management, especially in agentic research workflows where the agent picks a plausible-sounding reference without re-checking the original paper.

Citation / Claim Audit Kit is a minimal open-source experiment testing whether a lightweight LLM-as-judge can catch this layer.

§2 How it works

The kit takes (claim_text, cited_work_metadata) pairs and asks an LLM judge to return a strict JSON verdict:

{
  "citation_exists": true,
  "support_verdict": "supports | partial | weak | contradicts | not_relevant | unavailable",
  "rationale": "1-3 sentences explaining the verdict",
  "evidence_excerpt": "what the cited work actually says/does"
}

The judge relies on its own knowledge of the cited works. When the cited work is unknown to the judge, it must return unavailable rather than hallucinate. The kit is deliberately small — one Python file, ~300 lines — so the failure modes are inspectable.

Verdict semantics:

• supports: the cited work directly establishes the claim.
• partial: the cited work establishes part of the claim, or a related but weaker version.
• weak: the cited work is in the same area but does not establish the claim.
• contradicts: the cited work argues against the claim.
• not_relevant: the citation is off-topic for the claim.
• unavailable: the judge does not have enough information about the cited work.

§3 First audit: 30% of sampled citations were weaker than they appeared

We ran the kit over 20 manually extracted citation pairs from recent arXiv papers in the LLM / agent space. The result: 6 of 20 (30%) received a supports-below verdict — 3 partial, 3 weak. Two illustrative examples:

Example 1. A paper claimed that “MoE layers replace the FFN in modern LLMs” and cited Vaswani et al. 2017 as evidence. Vaswani 2017 is the original Transformer paper. It introduces the FFN as part of the standard architecture; it does not establish or even mention any MoE-replaces-FFN claim. The citation is real, the paper is real, but the citation is doing different work than the body text implies. The judge flagged this as weak.

Example 2. A claim about MoE offloading cited a routed-scaling-laws paper. The cited paper is in the right neighborhood — it studies expert count and capability — but the body text used it to support a claim about weight offloading specifically, which is a different operational concern that the cited paper does not address. The judge flagged this as partial: the cite is relevant to the broader topic but does not establish the specific claim made.

These are exactly the failure modes that pure existence-checkers miss. The citation exists, the paper exists, and the area is correct — but the citation does not establish what the prose says it does.

§4 Why this matters now

Agentic research pipelines are starting to produce papers at scale. The unit cost of generating a plausible-looking citation is approaching zero, while the unit cost of verifying that the citation actually supports the claim has not changed at all. The asymmetry favors citation noise.

A lightweight, embeddable audit step — one that runs on (claim, cite) pairs during paper drafting or review — could keep this noise out of the literature without requiring full-text fetching or heavy infrastructure. Citation / Claim Audit Kit is a first artifact in that direction.

The kit is intentionally a starting point. Three obvious extensions:

• Fetch the cited paper's abstract / introduction rather than relying on judge world-knowledge. This handles cited works the judge does not know.
• Integrate into review pipelines so the audit runs as part of paper review, flagging weak citations before publication.
• Run at scale on a recent slice of arXiv to characterize the base rate of weak / tangential / mischaracterized citations in published work.

§5 Try it

The code is available at github.com/t46/citation-claim-audit-kit under MIT. Issues, PRs, and counter-examples (citations the judge mis-graded in either direction) are welcome.

This is part of a wider exploration on what lightweight, agent-friendly infrastructure for research process integrity could look like. A companion artifact, Research Ledger Lite, captures the research process as a Markdown + Git + SQLite append-only ledger.