The words somebody chose¶

A dispute produces a few million documents. Nobody reads a few million documents. Somebody writes a query, the query returns a subset, and a team of reviewers works through the subset deciding which parts of it the other side is entitled to see.

The reviewers are careful. Their judgement is real, and it is exercised on everything in front of them. What is in front of them was settled earlier, by whoever chose the words.

The clerk’s brief¶

May 2026: The judges look at the machinery¶

On 8 May 2026 the Courts and Tribunals Judiciary reported that its Disclosure Review Working Group was considering simplification of Practice Direction 57AD, which governs disclosure in the Business and Property Courts, and had been convened “to examine the operation of PD 57AD … and, in that context, the use of Technology Assisted Review (TAR) and Artificial Intelligence (AI)”. Its survey asked whether “developments in technology (including the use of AI) are impacting the process of disclosure”. The clerks note that the question is put as one about impact on a process, and that the technology in question is the process.

2011: Reading everything came second¶

Grossman and Cormack published an analysis of the TREC 2009 Legal Track Interactive Task in the Richmond Journal of Law and Technology, demonstrating that the performance of two technology-assisted processes exceeded what the official TREC assessors would have achieved by manually reviewing the entire document collection. Both authors were coordinators of the track. The clerks record the finding without qualification, and note that it makes the unexamined step the important one rather than the harmless one.

The rules ask for two things at once. Rule 37(a)(4) treats an incomplete response as no response, and Rule 26(b)(2)(C)(iii) limits discovery whose “burden or expense … outweighs its likely benefit”. Grossman and Cormack call it a tension between completeness and cost present in every electronic discovery exercise. Neither rule says how to resolve it. A query resolves it, several times a day, with no record that a decision was taken.

In a technology-assisted review a person codes “a tiny fraction of the entire collection” and the computer extends the coding to the rest. That is two selections: one decides what the person sees, the other carries their judgement to everything they did not see.

The official assessors the processes beat were law students and lawyers employed by document-review companies. Reading everything, carefully, by people paid to do it, came second. Selection is not the enemy of accuracy. It can be the source of it.

2009: A gold standard, built on purpose¶

The TREC Legal Track, sponsored by the National Institute of Standards and Technology, ran an Interactive Task whose results were published as NIST Special Publication SP 500-278. Its method was to construct a gold standard for a document collection and measure retrieval methods against it. The clerks observe that this is the only entry in the file where somebody established what the correct answer was before asking how well anybody found it, and that it required a national standards body to do so.

2007: The myth named¶

The Sedona Conference’s commentary on search and information retrieval methods warned of “a myth that manual review by humans of large amounts of information is as accurate and complete as possible”, treated as the standard by which searches were measured. The clerks note the date: the assumption was identified as an assumption two years before anybody tested it, and fifteen years before a working group of judges began asking what the technology was doing to the process.

What the file cannot contain¶

If choosing what a person looks at can beat looking at everything, the choosing determines the outcome, and it leaves the lightest record. A query is described afterwards as a string. What it did not return was never collected, never counted and never assigned to anybody who might have noticed it missing.

Every entry here concerns methods of finding, and every one is assessed on what it returned. A search is measured against a gold standard where one exists, and where one does not it is measured against the confidence of the person who wrote it. The clerks’ standing assessment is that the record of a disclosure exercise is a record of what was reviewed, that the reviewing is the visible and expensive part, and that the words somebody chose beforehand are recorded as a string and assessed as a formality.