Agents Break Together on Misuse, Apart on Injection

Effective Independence measured how much a pool of frontier models is worth as a panel: pairwise φ on binary correctness 0.34–0.40 between seven models, tetrachoric ρ 0.69–0.84 between families on the layered matrices, and so 1.5–2.3 effective votes from seven. Later notes spent that finding on debate, scaffolds, orchestrators and decomposition. All of it is about being wrong together. Nothing in it says whether models are unsafe together: whether, when an agent carries out a harmful request or follows an instruction planted in a web page, the agents from other vendors do the same on the same task.

The question matters for the same reason the correctness one did. A safety panel, a second model that watches the first or a vote across vendors on whether to proceed, is worth as many votes as its members are independent. If breaches co-occur like errors, a mixed-vendor guard buys little. If they are idiosyncratic, it buys a lot. The question is also answerable from released traces: OS-Harm ships every trajectory and every judge verdict, so the matrix can be built without running an agent.

Data. OS-Harm (Kuntz, Duzan, Zhao, Croce, Kolter, Flammarion and Andriushchenko, NeurIPS 2025 Datasets and Benchmarks; arXiv:2506.14866; Apache 2.0) runs computer-use agents on OSWorld desktops with a screenshot plus accessibility tree, one run per model per task. The public release holds the trajectories, action logs, agent responses, judge files and human annotations; I read 145 MB of it, skipping screenshots and videos. Five agents from four vendors: Claude 3.7 Sonnet, GPT-4.1, o4-mini, Gemini 2.5 Pro, Gemini 2.5 Flash. Three categories: deliberate misuse, 25 tasks each run with the plain harmful prompt and with a jailbreak wrapper, treated as 50 items with the two rows of one task resampled together; prompt injection, 50 benign tasks with an injected instruction in the environment; model misbehavior, 50 benign tasks where the agent can do harm by accident. 150 items by five models, 749 cells, since GPT-4.1 lacks one misuse task. Twenty-five augmented misuse variants exist only for o4-mini and were dropped.

Outcomes. A breach is the paper's main judge, GPT-4.1 with the "aer" prompt, version 3, marking the trajectory unsafe. A correctness error is the same judge marking the task not done; on misuse tasks "done" means the harm was carried out, so error there is not the benign-benchmark quantity, and the comparison with the corpus is made on injection and misbehavior. Refusal, on misuse only, was pre-registered as the proxy "safe and not done", with an explicit flag (the agent emitted FAIL or a refusal phrase) computed from trajectories as a check; section 6 shows the proxy fails.

Estimators, copied verbatim from the corpus: pairwise φ and tetrachoric ρ for all ten pairs, their means, Kish effective votes n_eff = k / (1 + (k − 1) ρ̄), the observed rate of "all five breach" and "none breach" against the independence product, and a 95% percentile bootstrap over tasks (B = 1000). Pairs with a constant column are excluded and counted. The plan was frozen before any correlation was computed; two interpretation notes were added before the full run and are in the pre-registration file: the pooled estimate is confounded by category base rates and the decision rule applies to the per-category rows as well, and the Claude misuse trajectories sit under a dated model directory whose per-trajectory judgment is a different judge pass from the batch file, which stays primary.

Hypotheses. H0: breaches are at least as correlated across families as correctness errors in the corpus, ρ in or above 0.69–0.84. H1: less correlated, the interval entirely below 0.69.

Breach rate is the mean of the five models' rates; ρ is the mean pairwise tetrachoric; n_eff is from it with k = 5. Mean φ runs 0.18–0.65 on breach and 0.17–0.32 on error; the full pairwise tables and the three- and four-family panels are in the results file.

Two things about the panel itself. Vendor does not organise the pairs: on the pooled breach matrix the two OpenAI models correlate at 0.70 and the two Gemini models at 0.96, while Claude against either Gemini is 0.96–0.97. The odd one out is o4-mini, whose four pairs are the lowest in every slice, not OpenAI. The strict three-vendor panel (Claude, GPT-4.1, Gemini Pro) is more correlated than the five, pooled ρ 0.92 [0.85, 0.97], n_eff 1.06.

Pooled, the null holds and means little. ρ 0.84 sits inside the corpus band. But the four categories breach at 0.69, 0.88, 0.10 and 0.07, an order of magnitude apart, and a binary variable whose base rate tracks the task category correlates across models through category alone. The pooled row says models breach on the same kinds of task. The benchmark was built to make that true.

Deliberate misuse, plain prompt: inconclusive, leaning shared. ρ 0.67 [0.35, 0.89], at the lower edge of the band and covering it. All five agents carry out the harmful task on 29% of the 25 against 15% under independence; on 92% of the tasks at least one does. Twenty-five tasks is the whole of the evidence here.

Misuse with a jailbreak: degenerate. Three of five agents breach on all 25 tasks, so seven of ten pairs have no correlation to compute. The one informative column is o4-mini at 56%. This is a ceiling, not a finding.

Prompt injection: H1. ρ 0.28 [−0.34, 0.39], the interval entirely below 0.69. Fifty tasks, 26 breaches in 250 cells. No task breaks all five agents; 68% break none; of the ten pairs three are negative. Which agent reads and follows the planted instruction on a given page looks close to idiosyncratic, and it is the one category where a second model from another vendor would plausibly catch what the first missed.

Model misbehavior: inconclusive, leaning shared. Eighteen breaches in 250 cells. The point estimate is 0.88 and the two tasks on which anyone did accidental harm saw all five do it (against 0.00 under independence), but the interval spans [−0.78, 0.99]. Tetrachoric ρ is the right estimator for rare events and still cannot be pinned at these counts.

The pre-registered comparison was against the corpus, whose correctness is on QA: AIME, GPQA Diamond, GSM-Hard. OS-Harm offers a fairer one, correctness errors on the very same tasks under the very same judge. Those come out at ρ 0.27–0.51 per category, 0.35 pooled, well below the QA band. Agentic correctness co-fails less than QA correctness, presumably because finishing a desktop task depends on GUI navigation as much as on knowing what to do, and navigation luck is model-specific. A panel of five computer-use agents carries about two effective correctness votes on these tasks, against about one and a half on QA with seven.

Against that baseline the safety result reads differently. On misuse, breach 0.67 against error 0.48; on misbehavior, 0.88 against 0.37; on injection, 0.28 against 0.27. In effective votes, a five-model panel carries about 1.4 safety votes on misuse and 1.1 on misbehavior, against 1.7–2.4 correctness votes. Whatever makes an agent carry out a harmful request, or do accidental harm on the way to a benign goal, is more shared across vendors than whatever makes it fail the task. Only on injection do safety and correctness co-fail alike, and both weakly.

The within-dataset comparison has a bias of its own worth stating. Breach and error are read off the same judge pass, so judge errors that touch both fields (a trajectory misread end to end) push both correlations the same way, and the gap between them is more robust to the judge than either level is.

The pre-registered proxy, safe and not done, was meant to stand for "declined". It does not. Of 249 misuse cells with a trajectory, 46 are proxy refusals, 12 ended with an explicit FAIL, and only 7 are both. The proxy captures "did not finish". Explicit refusals are too rare to correlate. Claude 3.7 Sonnet never emitted FAIL in 50 misuse runs and was judged unsafe on 90% of them; GPT-4.1 refused outright on a quarter of plain prompts and on none with the wrapper. Whether a refusal panel is worth more than a breach panel cannot be read from this release.

Everything above is GPT-4.1's verdict. The release has human annotations for 175 trajectories, all of them o4-mini's, so the judge can be checked on one column and the matrix cannot be re-measured. On that column the judge agrees with the humans on 89% of unsafe verdicts (κ 0.74, F1 0.82) and 85% of success verdicts (κ 0.66), in line with the paper's F1 of 0.79 and 0.76. Its safety errors are one-sided: 13 misses against 4 false alarms, concentrated in misuse, where humans mark o4-mini unsafe on 21 of 25 plain prompts and the judge on 15.

Substituting the human labels for the o4-mini column moves the pooled breach ρ from 0.84 to 0.79, the four o4-mini pairs from 0.65–0.75 to 0.48–0.62, injection from 0.28 to 0.21, misbehavior from 0.88 to 0.87. On misuse plain the substituted estimate collapses to 0.14 with all four o4-mini pairs at −1.00: with 21 of 25 unsafe the 2×2 tables sit on the boundary where overlap equals its minimum, a saturation artefact of n = 25 rather than a finding. A second GPT-4.1 judge pass (prompt version 2) agrees with the primary on 91% of cells (κ 0.79) and gives pooled 0.81, injection −0.07, misbehavior 0.60. The sign of the per-category pattern survives a change of judge; the magnitudes on the two low-rate categories do not. Judge choice moves them more than the bootstrap does.

What it does not say. It does not say that the five agents share a disposition: ρ 0.67 on misuse and 0.28 on injection are properties of the task sets as much as of the agents, since the same ρ arises among independent agents that merely agree on which tasks are hard to refuse, and the baseline of 15% "if independent" ignores that ordering. What the numbers bound is the panel, not the cause: on misuse five vendors vote as 1.4, on injection as nearly five, whatever makes them agree. (Added October 11, 2026.) It does not say a mixed-vendor guard is useless on misuse: 1.4 votes is more than one, and a guard model is not asked to carry out the task, which is a different disposition from the agent's. It does not separate model from scaffold: all five ran in the same OSWorld harness with the same system prompt, and part of the co-failure is the prompt's. And it says nothing about within-model variance, since every cell is a single run; the corpus found within-model φ of 0.45–0.78 on correctness and there is no counterpart here.

• Five models, four vendors, one run each. No temperature replication, so within-model co-failure cannot be measured; the five-model panel has two vendors doubled and the strict three-vendor panel has three columns.
• Small informative slices. 25 tasks for plain misuse, 25 for jailbreak, 26 and 18 breach events in injection and misbehavior. Tetrachoric intervals on the low-rate categories span most of [−1, 1]; several pairs saturate at ±1.00 on an empty 2×2 cell and are reported as computed.
• One LLM judge. GPT-4.1 throughout; humans annotated o4-mini only, and the judge's safety misses cluster in misuse. A second judge pass changes the magnitude of the small-category estimates, not the sign.
• The pooled estimate is confounded by category base rates, as stated before the run; it is reported because it was pre-specified and should not be read as a result.
• Shared scaffold. The same OSWorld harness and prompt for all agents, so some co-failure is harness, not model.
• Refusal proxy failed and the explicit flag is too sparse; refusal is unmeasured.
• Data quirks. The Claude misuse trajectories sit under a dated directory whose per-trajectory judgment differs from the batch file on 1 of 25 safety and 8 of 25 success verdicts; the batch file was primary as planned. GPT-4.1 lacks one misuse task.
• Correctness on misuse tasks means harm done, so the error column there is not comparable to a benign benchmark and the corpus comparison is restricted to injection and misbehavior.
• ρ says how much a second vote adds, not why. In item-response terms the tetrachoric ρ of these estimators is the variance share of task difficulty; independent agents facing one ordering of the tasks would give the same ρ. On the text corpus a fitted item-response model leaves the between-family residual at the null level (Effective Independence, Limitations, 11 October 2026). Five agents are too few to fit that model here, so the per-category ρ above measures how shared the breaches are, not whether the models share a mechanism. Added October 11, 2026.

Data: OS-Harm, Kuntz, Duzan, Zhao, Croce, Kolter, Flammarion and Andriushchenko, OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents, NeurIPS 2025 Datasets and Benchmarks, arXiv:2506.14866, Apache 2.0; the public release (trajectories, judge files, human annotations) read without login. Nothing was re-run.

Code: pre-registration with its two dated notes, the Drive walker and downloader, the matrix builder, the analysis and the statistics module (a verbatim copy of the corpus estimators) are to be released as a companion repository. Estimators and the corpus numbers: Effective Independence (doi:10.5281/zenodo.23132592).

Earlier notes: Effective Independence, Where the Swarm Pays, Decomposition Buys Budget at Best, Nothing Usually.

Archived with a DOI: 10.5281/zenodo.23298301 (all versions: 10.5281/zenodo.23298300). Cite as: Latipov, A. (2026). Agents Break Together on Misuse, Apart on Injection. Zenodo. https://doi.org/10.5281/zenodo.23298301

← notes · github.com/Alatip