What counts as evidence

1. Why this page exists

A tool that says "verified" without saying against what is asking to be trusted blindly. Assay does not ask for that. Every verdict names the comparison it made, and this page is the definition behind those words.

2. What counts as evidence

Assay uses "evidence" in a narrow sense. Something is evidence only if all four hold:

1. A person could open and read it. It is a record or a text, not a feeling. Anyone who disagrees with a verdict can open the same record.
2. It comes from a source named on this page, whose coverage and limits are stated, so the meaning of "found" and "not found" is knowable.
3. It was fetched or supplied at the time of the check. An old answer is only used when it is plainly marked stale.
4. It can be compared with the text mechanically, so that two people running the check get the same result.

What does not count, anywhere in Assay: a model's statement that a source exists or says something; a tag the model wrote (such as [[c3]]) before it has been checked against the claims you approved; a search engine's snippet; a link elsewhere in the same text; and the fact that a web page loads, which is evidence of liveness only and is reported as that. A model's judgement that a source supports a sentence is not evidence either: only a quotation it points to can be, and only once that quotation has been found in the source.

Why these rules. The documented harms from AI-written text share one pattern: something plausible was accepted because nothing it could be compared with was looked at. Four conditions make the comparison contestable (open it yourself), bounded (the source's limits are written down), fresh, and repeatable.

3. Evidence, check by check

4. The sources

These are all the places Assay ever sends a request, apart from the addresses in a text you ask its link checker to check. It does not add to this list by itself: a new source is a new thing whose answers are believed, so adding one is a decision for a person (see section 8). The self-test column is the work each source must be able to find before its "not found" is believed.

5. Source quality: the tiers behind the gate

The gate judges how a claim is sourced, not whether it is true. Sources are ranked by tier, from a policy that you supply (Assay ships only a small neutral default).

A rule says which claims need stronger backing, chosen by label or by what the text reads like, so a wrong label does not dodge it, and satisfied by alternatives counted over different sites ("one authority, or two news sources"). Nobody, including the claim's author, can raise an unlisted domain above tier 3 by saying so.

6. How a verdict is reached

Every item in every check gets one bucket and a sentence in plain words.

The rule that matters most: "could not look" is never "not found." If a source is throttled, down, or answers in a shape it does not normally use, the result is check ("this says nothing about the citation"), never failed.

7. How the safeguards work, and what they do not do

Assay is only as trustworthy as its safeguards, so be clear about what they are:

• They regulate themselves while Assay runs. A failing or throttled source is left alone and probed later; a changed reply is treated as an outage; a source that cannot find a work it must know has its "not found" discarded; an earlier answer is served, marked stale, when sources are down.
• They do not update themselves. Nothing in the running library notices that a source has permanently renamed a field or retired the work used as its self-test. In those cases Assay fails safe (the source is treated as unreliable and its answers are discarded) but does not repair itself, and a source that stays broken quietly reduces coverage.
• Something outside the library notices. assay health asks every built-in source for a work it must know, and a scheduled job in the repository runs it every week and opens an issue when one answers wrongly. That is monitoring, not repair: a person still has to fix the source, and a weekly check can miss a change for up to a week.

Self-regulating, then, not self-updating. The technical paper sets out the mechanisms and their failure modes.

8. Adding your own source

You can add a source; Assay will not add one to itself. A library that adopted sources automatically could be pointed at a poisoned one, and would change which hosts it contacts without anyone deciding to. A source you add inherits the same safeguards: its replies are shape-checked, its "not found" is not believed until it passes its self-test, and a failing source is left alone.

You can write one with data, for any service that answers JSON over https. It must have a fixed host and a self-test work, or Assay refuses it:

import { createAssay, defaultSources, jsonSource } from '@extrafemi-ai/assay';

const myIndex = jsonSource({
  name: 'My Institution Repository',
  url: 'https://repo.example.org/api/search?title={title}&author={surname}',
  results: 'data.items',
  fields: { title: 'name', authors: 'writers', year: 'meta.year', doi: 'ids.doi' },
  authorName: 'fullName',
  canary: { title: 'A work this repository certainly holds', surname: 'Surname', authors: ['Surname'] },
});
const assay = createAssay({ sources: [...defaultSources, myIndex] });

Or implement the SearchSource or ValueSource interface in code, or supply figures you checked yourself with seededSource. To add a source to Assay itself, open a pull request with the source, a fixture recorded from the real service, a self-test work, and a note saying whether it was verified live. The reference has the details.

9. Known limitations

• Evidence that something exists is not evidence that it says what the sentence claims. A verified citation can still be attached to a false claim. The support check reaches toward this, but only as far as a model's reading plus a quotation found in the source.
• Coverage has gaps. Books, reports, grey literature and non-English work are thinly indexed, so a real work can come back "not verified". That is why Assay says "not verified", never "invented".
• Sources can be wrong. A bibliographic database with an error will pass it on.
• "No notice recorded" is not proof. For retractions, notices arrive late and from several places.
• The labelled sample is small. Thirty hand-labelled citations are not a benchmark, and no accuracy figure is claimed. See what Assay cannot do.

10. Citing Assay, and the licence

Assay is free and open source under the MIT licence. To cite the method:

Olorunfemi, O., & Olusegun, P. (2026). Assay: Checking AI-written text against evidence, one question at a time (Technical report, version 1.0). Extrafemi Research. https://assay.extrafemi.com/paper/

11. Corrections

If a source is described wrongly, a limit is missing, or a definition here does not match what Assay does, open an issue on GitHub. This page is generated in part from the code: the self-test works in section 4 are read from the sources themselves.