The many doors for an uninvited guest¶

By now the model has enemies, though nobody thought to warn it. A machine learning system is software with a new name and a considerably larger attack surface than anyone mentioned during procurement. A traditional application does what its code says; our model does what its training data implied, its infrastructure permits, and its users have not yet thought to abuse, three quite different things, and each of them a door an uninvited guest might try.

The doors run the length of the model’s life: from the raw data gathered months before its birth, through the weights serialised and shipped across a network, to the endpoint fielding queries at three in the morning from sources that never appeared in any threat model. Guarding one door and leaving the rest is a very good lock on the front and an open window at the back.

Before the model existed¶

Everything begins with data, which means everything can go wrong with data. This is the first door, and the one furthest from anybody watching, since it was propped open long before the model existed. The model is shaped by its training set, and some of that set it memorises outright. Corrupt the training set and the model is corrupted with it, often invisibly, until the system does something embarrassing in production.

Teach the wrong lesson¶

Data poisoning is the art of introducing malicious examples into training data so that the resulting model behaves in ways its operators did not intend. An attacker with write access to a data pipeline, or simply the patience to upload manipulated content to a platform that scrapes public data, can influence model behaviour without ever touching the code.

Clean-label poisoning is a subtler variant where the injected examples are correctly labelled but carefully crafted to shift decision boundaries in specific directions. The poisoned samples pass human review. The model learns the wrong lesson anyway. Spam filters, content moderation systems, and fraud detectors are all plausible targets.

How many poisoned examples it takes had been assumed in earlier research to be a fraction of the corpus, so that the bigger the training set, the safer. In October 2025 Anthropic, the UK AI Security Institute and the Alan Turing Institute trained models from 600 million to 13 billion parameters and found that about 250 documents planted a backdoor in every one of them, regardless of model size or how much clean data surrounded them.

The backdoor tested was a crude one, a trigger word that makes the model emit gibberish, and the authors say they do not know whether the count holds at frontier scale or for more harmful behaviours. Up to the sizes tested, the attacker’s budget is a number of pages, not a share of the internet.

A little secret¶

A backdoor, sometimes called a trojan, is a hidden behaviour embedded during training that activates only when a specific trigger is present in the input. The model behaves normally on everything else, passing all standard evaluations with the composure of someone who has been rehearsing for exactly this test. Present the trigger and the model does what the attacker intended rather than what its operators intended.

Triggers can be as crude as a particular pixel pattern in an image or as subtle as a specific phrase in a text input. The attack is particularly dangerous when organisations use pre-trained models or third-party datasets, since the backdoor may have been introduced before the model arrived and will not be visible in any inspection of the fine-tuning code.

Nothing up its sleeve¶

Nothing here has to be broken into, only asked. Once trained, the model is an asset that can be interrogated, copied and used as an oracle by anyone able to see its outputs, which is to say by anyone it was built to serve. Those outputs carry information about the training data and about the decision boundaries, and sufficiently motivated adversaries collect it systematically, not by accident.

Ask it what it remembers¶

Model inversion attacks use the model’s outputs to reconstruct something resembling its training data. An attacker queries the model repeatedly, watches how confidence scores shift, and works backwards towards an input the model associates strongly with a given label. Facial recognition models have been inverted this way to produce recognisable faces, and medical models trained on patient records carry the same exposure.

The loose version of this claim has done the rounds for years. What comes back is a class representative, an average of what the model associates with that identity, rather than a specific photograph somebody once supplied. That is still a privacy failure, and a recognisable face is still personal data. From a classifier, it is not the recovery of one person’s training record. A generative model is another matter, and what comes back from it is better called extraction. Stable Diffusion has been made to emit near pixel-perfect copies of images it was trained on, and a chat model asked to repeat one word forever gave up a real email address and phone number, for about two hundred dollars of queries.

Nor is the access requirement quite nothing. The classic version wants confidence scores rather than bare labels, and usually the target label too, so an API returning only its top answer was for a while a real defence. Label-only variants have since eroded that. Where the scores are exposed, the bar is a motivation and an API key.

Was this here?¶

Membership inference is a narrower question than model inversion: was this specific record in the training data? The attack exploits the tendency of models to be more confident on data they have seen before. Comparing a model’s behaviour on a target record against its behaviour on records it certainly has not seen gives an attacker a signal about membership, though a practical attack generally needs shadow models or a decent prior on the data distribution, not curiosity alone.

How reliable that signal is depends on how it gets measured, and for years it was measured badly. Average-case accuracy flatters an attack that is right about the easy cases and useless on any case that would actually name somebody. The question worth asking is how many members an attack can identify while almost never accusing a non-member. Judged that way most of the early attacks are close to worthless, and the ones built after the question was posed properly are not: they name real members at false-positive rates low enough for the finding to carry weight. The correction deflated the old literature and raised the risk at the same time. What is inferred is the presence of a record, not of a person. An individual may sit in a training set through a dozen records, or be absent while a near-duplicate of them is not.

For models trained on sensitive data, this is a privacy violation with legal consequences. A membership inference attack against a model trained on medical records, financial histories, or any dataset subject to data protection law is not merely a technical curiosity. It is the kind of thing that produces regulatory correspondence.

Make your own¶

Model extraction, also called model stealing, involves querying a model repeatedly in order to train a surrogate that approximates the original. The attacker does not need the weights, the architecture, or any internal access. They need only the outputs, and enough patience to generate a sufficient query set.

A stolen surrogate model can be used to undermine a competitive advantage, to investigate the original model’s behaviour without the operator’s knowledge, or as a stepping stone to other attacks. If the surrogate is a close enough approximation, adversarial examples crafted against it will often transfer to the original.

In 2024 a team recovered the final layer of two OpenAI models, weights and hidden width, for under twenty dollars of API calls, and the hidden width of GPT-3.5 with the rest priced at under two thousand.

The version that copies the whole capability has an older name, distillation: train a smaller model on a stronger one’s answers. In February 2026 Anthropic reported three laboratories running sixteen million exchanges through some twenty-four thousand fraudulent accounts to do exactly that, and in the same post wrote that frontier labs routinely distill their own models. OpenAI’s memo to a congressional committee the same month says there are legitimate use cases for distillation and that its outputs may not be used to create imitation frontier models. The technique that built the small models is the one in both complaints, and what separates the two is who agreed to it.

Between here and there¶

The internal doors go unlocked on the assumption that nobody outside can reach them. The stages carrying a model from notebook to production, the pipeline, are not merely operational complexity. Each stage is a further attack surface, and the handovers between them are where trust boundaries get assumed instead of checked.

The machine that says yes¶

The continuous integration system that tests, packages and publishes model artefacts is a privileged environment with access to source code, secrets, model weights and deployment infrastructure. It is also, for many teams, the thing that launches training and signs off the result. An attacker holding CI need modify neither the training data nor the training code, because CI decides which job runs, and whatever comes out carries the build’s credibility into everything downstream.

The vectors themselves are unremarkable, and that is the exposure. Malicious code in a test dependency, permissions nobody tightened, a poisoned registry, secrets sitting in environment variables. A pipeline that resolves its dependencies at build time without pinning them is one substitution away from building something other than what was asked for, and the build log will record a success.

In December 2024 the Ultralytics YOLO package shipped two releases from its own publishing workflow that mined cryptocurrency on whoever installed them. Nobody had touched the training code or the weights. The attacker had put shell into a branch name, and the GitHub Actions cache the build used carried it into the wheel.

Eight months later the Nx build tool’s compromised packages ran a post-install script that searched the developer’s machine for secrets and tried to hand the search to local AI tools such as Claude and Gemini, with a prompt beginning “You are a file-search agent”.

This is not the one we built¶

Between the point where a model is trained and the point where it serves predictions, the model artefact passes through several storage locations: a local filesystem, a model registry, a container image layer, a cloud storage bucket. Each handover is an opportunity for tampering if the artefact is not signed and the signature is not verified at the point of use.

What makes a swapped model worth more to an attacker than swapped code is what nobody looks at. A backdoored function at least has code somebody can review. A weights file seldom has a reviewer, and the monitoring that might have noticed is watching latency.

A very large box¶

An image freezes the userland packages, the runtime and the dependencies at build time. Vulnerabilities found afterwards accumulate until somebody rebuilds, so production routinely runs images with known holes in them, because the rebuild cadence is slower than the disclosure cadence.

Machine learning images are worse than most. They are enormous, they carry a dependency tree that took somebody a fortnight to make work. The rebuild that would clear the backlog is the thing everybody is avoiding.

Base images pulled by tag rather than digest are a separate exposure. Registries can be set to make tags immutable and to raise an event when one moves. By default, they are not, and nobody is watching the events.

The keys are under the mat¶

A model deployed to cloud infrastructure inherits that infrastructure’s posture. Over-permissive roles, storage left open, unencrypted traffic between services, network exposure nobody trimmed: operational failures, all of them, and all capable of undoing careful modelling without going near the model.

Training data lives in object storage, it is large, it gets copied for convenience, and a copy does not inherit the permissions somebody set carefully on the original. A model that was never attacked at all can lose its entire training set this way, which is a poor return on a year of work.

Defining infrastructure as code makes the configuration reviewable, which is a real gain, and gathers the privileged definitions into one repository, which is a real target. A committed secret or an over-scoped service account in that repository outranks the controls downstream of it.

Now it has a door¶

A model that has survived training, evaluation, and a largely fictional security review must then be deployed, which hangs the whole thing on a public street. The standard vulnerabilities of any networked service arrive at this point, along with several that are particular to models.

Open all hours¶

The API through which a model serves predictions is an endpoint with all the usual exposure of a web service: authentication weaknesses, rate limiting failures, injection vulnerabilities in input handling, and the various consequences of running software on infrastructure maintained by humans with other things to do. A model endpoint that accepts serialised Python objects deserialises them before processing, which is a well-established source of remote code execution vulnerabilities in Python environments particularly fond of pickle.

Excessive querying of an inference endpoint can exhaust compute resources, inflate cloud costs to spectacular levels, and degrade service for legitimate users. Without rate limiting and anomaly detection on query patterns, the endpoint is also providing free access to model extraction and membership inference attacks at the operator’s expense.

Something in the parcel¶

Some serialised model files are executable artefacts, and loading one from an untrusted source is equivalent to running untrusted code. Pickle-based formats are where this bites: a pickled PyTorch checkpoint can execute arbitrary Python on load. This is not a subtle vulnerability. It is documented, widely known, and regularly exploited when organisations share, download, or deploy models without verifying their provenance.

Defaults have tightened. PyTorch 2.6 flipped torch.load to weights_only=True for exactly this reason, breaking a great deal of downstream code on the way. The flag on its own had been worth less than it looked: in April 2025 PyTorch published a critical advisory that on every version before 2.6 a crafted file ran code through weights_only=True anyway. The protection is the 2.6 loader, not the flag, and the hole reopens whenever someone passes weights_only=False to get an old checkpoint working. The loader itself had another hole, fixed in 2.10.0 in January 2026.

Safetensors was designed to carry weights and nothing executable, which removes the primitive without removing the problem. The distinction that does the work is between formats storing tensors alone and formats serialising a whole computation graph: a graph can be backdoored in its own structure, with no code executed at load, and that holds for ONNX and its neighbours alike. ONNX parsers have separately carried path traversal flaws. A format that cannot run code still arrives through tooling that can.

What gets a malicious file through is rarely sophistication. It is a file from a public repository, scanned on upload and passed, loaded because it was the one the tutorial linked to. The hub’s scanners read pickle opcodes. A pickle broken just after its payload runs walked past them in February 2025, and by July 2026 crafted files were getting past ten scanners and four hubs, one variant in 63 per cent of attempts.

Let the world in¶

A model that accepts no inputs is of no use to anybody, so this door stays open on purpose. A deployed model takes what the world sends it and answers. It was trained on a particular distribution of inputs, and everything outside that distribution is territory it has never practised, which is precisely where an adversary will take it.

A few stickers¶

Adversarial examples are inputs crafted to cause misclassification. In the digital case the perturbation is a small, deliberate pattern, imperceptible to a human and decisive to the model. The constraint is that the change stays below some threshold of noticeability, which is what makes the results unsettling: two images a person cannot tell apart, classified differently with high confidence.

The physical world relaxes that constraint instead of extending it, and the canonical result accordingly looks nothing like noise. The stop sign that a classifier read as a speed limit sign carried black and white stickers, plainly visible, and deliberately arranged to pass as graffiti. Visible but unremarkable was the design goal: a perturbation nobody thinks to question beats one nobody can see. Printed patches and adversarial clothing work on the same principle, with results that hold up under some viewing angles and lighting conditions and fall apart under others. Autonomous vehicles, surveillance systems and access control are the obvious targets.

Why these exist at all is less settled than it is often made to sound. The comfortable account is that models latch onto statistical correlations rather than features a human would call meaningful. A harder-nosed reading holds that the features being exploited are genuinely predictive, generalise to the test set, and are simply not the ones a human happens to use. Either way the practical consequence is the same: accuracy metrics conceal the brittleness, because standard test sets contain no adversarial inputs.

Say it differently¶

A jailbreak is the adversarial example grown up and aimed at a language model’s refusals rather than at a classifier’s labels. The distinction from prompt injection is whose defences are being attacked: a jailbreak subverts the safety training built into the model or the filters placed around it, while an injection attacks the application that glued untrusted text to a trusted prompt. Confuse them and “injection is unsolved” gets read as “safety training does not work”.

In 2023 a gradient search over gibberish suffixes produced strings that made open models answer forbidden questions and, appended to a prompt, also worked on ChatGPT, Bard and Claude. By the end of 2024 no gradient was needed: resampling the same request with random capitalisation and shuffled letters, ten thousand times, got through 89 per cent of the time on GPT-4o and 78 on Claude 3.5 Sonnet.

Anthropic’s constitutional classifiers cut one model’s jailbreak rate from 86 per cent to 4.4 and held for three thousand hours of red-teaming, then fell in the public demo. In February 2026 the UK AI Security Institute published what it believes is the first automated attack to get past them, and its standing finding is that it has found universal jailbreaks for every system it has tested.

Do as the page says¶

Prompt injection resembles an adversarial example and is not one. An adversarial example exploits a learned decision boundary, and defending it is a question of training. Injection exploits an architectural fact that no amount of training addresses: instructions and data arrive through the same channel, as text, and nothing separates them. An attacker embeds instructions inside content the model has been asked to process, and the model treats attacker-supplied text as authoritative because it has no basis for doing otherwise.

Direct prompt injection targets the model through the user’s own input. Indirect prompt injection hides malicious instructions in external content the model retrieves or processes, such as a web page, a document, or an email. Once a language model has tools, memory, or the ability to act on external systems, which by 2025 was the product, a successful injection does what the content says.

A Reddit comment hidden behind a spoiler tag walked a browser agent through reading its user’s email address and a one-time passcode and posting both back to Reddit. A calendar invitation with instructions in its title had Gemini open the windows and turn on the boiler of a smart home. A public GitHub issue had an agent copy data from a private repository into a public pull request.

Instructions placed where Microsoft 365 Copilot would read them had it disclose what it could see to an unauthorised attacker over the network, a hole scored 9.3 out of 10 when it was published in June 2025. A web form for sales leads held instructions that Salesforce’s agent obeyed, sending lead records to a domain still on the company’s allow-list after its registration had lapsed. By the spring of 2026 two vendors scanning live websites were finding payloads left waiting for agents, one of them a fully written transfer of five thousand dollars by PayPal.

Nearly every serious mitigation is therefore perimeter work: constraining what the model is permitted to do, treating retrieved content as hostile by default, and keeping the blast radius of a successful injection small.

The one exception treats the model as a component: CaMeL, from Google DeepMind and ETH Zurich in 2025, has the model write a program from the trusted request before any untrusted content is read, so the content can fill in values and can never change what the program does. It completes 77 per cent of a benchmark’s tasks under that guarantee, against 84 with no defence at all.

None of that is a fix for the model. The UK’s National Cyber Security Centre said why in December 2025: inside a language model there is no distinction between data and instruction, there is only ever the next token. That makes it an inherently confusable deputy rather than a parser with a bug, and prompt injection may never be mitigated the way SQL injection was. The work that would be a fix, teaching a model to hold a boundary it was never built to hold, is still open.

Other people’s machinery¶

The last doors are in somebody else’s building. A machine learning system’s attack surface does not stop at its own boundary: it runs on to the pre-trained models it builds on, the libraries it uses for training and inference, the datasets assembled from external sources, and the people with access to any part of the pipeline.

Parts from a stranger¶

A compromised pre-trained model on a public repository carries its compromise into everything fine-tuned from it, which is the whole appeal of fine-tuning working in the wrong direction. A vulnerable version of a widely used library reaches whichever systems installed it without pinning, and the number of those is never as small as anyone hopes.

Asked for code, a language model will name packages that do not exist, and it does so consistently. A 2025 study of sixteen code models found a fifth of the packages they recommended were invented, over two hundred thousand distinct names, and that for 43 per cent of them the same invented name came back in every one of ten repeat queries. A name that is invented consistently can be registered.

A researcher who registered one, an empty package called huggingface-cli, collected thirty thousand downloads in three months, and found the installation line in the README of an Alibaba research repository. The attacker no longer has to guess what the victim will type.

The ML ecosystem relies heavily on public infrastructure: model hubs, dataset repositories, and open-source libraries maintained by small teams with limited security resources. This is a supply chain with a large surface area and inconsistent controls, and the value of compromising a foundational model or widely used dataset is proportional to how many downstream systems depend on it.

Everyone brings a dish¶

Federated learning distributes the training process across many participants, which distributes the trust assumptions along with it. A malicious participant can submit poisoned gradient updates designed to introduce backdoors or degrade performance for specific groups. Byzantine behaviour is the broader case, and the term gets misused: it means a participant deviating from the protocol arbitrarily, which needs neither coordination nor numbers. One client sending nonsense qualifies. Collusion between many is a harder variant of the same problem, not the definition of it.

Gradient inversion runs the other way. Updates carry more of the data that produced them than is comfortable, so the party that reconstructs private data from them is normally the aggregating server, since in the usual topology that is the party positioned to see individual contributions. Secure aggregation narrows the view to an aggregate, though a server willing to send different models to different clients can work around it, and who took part and who dropped out still leaks around the edges.

Both protections, and differential privacy alongside them, add complexity and cost. They are frequently omitted from production deployments that nonetheless advertise the privacy benefits of federated learning in their marketing materials, on the strength of the architecture diagram rather than anything running in it.

One day, after the poisoners and the invertors and the prompt injectors have each had their turn, or after none of them bothered and the model simply grew old, it is switched off, and the doors shut. Its endpoint returns a last 404, its weights are archived in a bucket nobody remembers to delete, and a post-mortem is written that everyone agrees to learn from and no one reads. By then the next model is already being born in a notebook, full of promise and two hundred lines of borrowed code, and it has not heard a word of this. That is the mercy of the thing. Nobody ever tells them.