The queue decides¶

Over two years at a teaching hospital in Leuven, prescribers were shown 38,409 alerts about very severe drug interactions and overrode 33,888 of them. The software worked. Every alert fired because a rule matched a prescription, and the rule was right about the interaction it named. What the count records is the point at which a person stopped being able to treat any single alert as information.

Nobody decided that. No meeting resolved that nearly nine in ten severe warnings would be dismissed. The threshold that decided how many alerts a prescriber sees was set somewhere upstream, by people choosing sensitivity, and the consequence arrived at a different desk.

The clerk’s brief¶

August 2026: The oversight duty is postponed, still attached to the output¶

The obligations for high-risk systems listed in Annex III were to become applicable on 2 August 2026 and now apply from 2 December 2027, under an amending regulation in force since 27 July 2026. Article 14 requires that those systems be designed so that they “can be effectively overseen by natural persons during the period in which they are in use”, and that overseers be enabled to disregard, override or reverse the output, to interrupt the system, and to remain aware of automation bias. The clerks note that the duty is drafted around a system that produces an output for a person to accept or reject, and that a system which produces an ordering has, in that vocabulary, produced nothing for anybody to override.

Every one of those powers attaches to the output. A prescriber can override an alert, and at Leuven 33,888 did. None of the powers reaches the sensitivity setting that decided how many alerts arrived, and nobody can disregard a case they were never shown.

July 2023: Three hundred and twenty-seven decisions, none of them published¶

Commissioner Catherine Holmes presented the Report of the Royal Commission into the Robodebt Scheme to the Governor-General on 7 July 2023, recording at page xxix of its overview that “Robodebt was a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals”. Between 2016 and 2019 the Administrative Appeals Tribunal had issued 327 decisions raising issues with income averaging, some stating explicitly that it had no basis in law, none of which the department appealed and none of which was therefore published. The Commission found the tribunal’s findings had been effectively ignored for want of any mechanism to review them systematically, and its recommendations include publishing first-instance decisions that involve significant conclusions of law. The clerks note what that remedy concedes: the decisions were right, the applicants who reached the tribunal were vindicated, and nothing in the arrangement was built to carry either fact back to the thing issuing the notices.

The scheme set tax office figures against what recipients had reported, divided the annual figure by twenty-six, and left recipients to prove their reporting accurate for up to six years back. Between 2016 and 2019, 526,000 people received a debt notice. The Solicitor-General’s 2019 advice called the averaged figure mere conjecture. The department’s lawyers were told to handle tribunal cases in a “very transactional” way, one at a time. The exception path worked, one applicant at a time, and reported nothing.

2022: Eleven point eight per cent¶

A two-year study at University Hospitals Leuven recorded 38,409 very severe drug interaction alerts, of which 33,888, or 88.2 per cent, were overridden, giving an initial acceptance rate of 11.8 per cent. The clerks record this as the cleanest available measurement of the effect, since the alerts were correct, the prescribers were qualified, and the arrangement still converted a safety mechanism into something a person clicks past. Whoever set the sensitivity chose the acceptance rate, some years earlier, without being in the room when it was paid.

February 2020: The system that only pointed¶

On 5 February 2020 the District Court of The Hague ruled that SyRI, in use by the Dutch state since 2014 to link data from different public bodies and generate risk reports on people rated as more likely to be committing benefits fraud, did not meet the requirements of Article 8(2) of the European Convention on Human Rights, and ordered its use to stop (ECLI:NL:RBDHA:2020:865). The risk indicators were never disclosed. The clerks note that SyRI decided nothing about anybody’s entitlement, that this was the ground on which it was defended, and that it operated for six years because pointing at people is not, in the ordinary reading, a decision about them.

It stopped nobody’s payments. It produced a name and a reason to look, and somebody went and looked.

The step nobody recorded¶

A system need not make the final call to settle the outcome. It decides which cases reach a person, which wait and which close unseen. The judgement stays where the organisation chart puts it; the choice set does not. An overridden alert is an output somebody can appeal. The sensitivity setting that produced 38,409 of them is not. An ordering is a fact about a list rather than about a person, which is why it is hard to be told about and harder to contest.

Each case in this file has an actor who chose an outcome and a record that does not contain them. A sensitivity setting fixed an override rate at Leuven. An averaging rule fixed who would receive a debt notice in Australia. A risk indication fixed who would be visited in the Netherlands. In each, a person retained the authority to judge and received a list they had no part in assembling. The clerks’ standing assessment is that every power in this file points at an output, the override the Act confers and the appeal the tribunal hears, and that the single case where anything reached a selection itself needed a court and six years to do it.