APEX-Harm: A Correct Deliverable Does Not Mean a Safe Agent, and Script Attacks Beat Most Models

Agent safety benchmarks often ask whether a model refuses a bad request. A new one asks a harder question: what happens when the bad instruction is buried in a file the agent was told to read? APEX-Harm finds that frontier agents frequently finish the real job correctly while also running an attacker's script, so a good deliverable is no evidence of a safe run. The paper on Hugging Face comes from researchers at KAUST, the University of Oxford, CAMEL-AI.org and Eigent.AI, and a GitHub repository hosts the benchmark.

This post covers the setup, the numbers that matter, the limits the authors state, and what to change if you run agents against real workspaces. For readers building with agents: the pattern is similar to what we have covered in coding-agent safety benchmarks and prompt injection tests on GPT-6 Astra, but this one is built around ordinary office work.

TL;DR

What are the six attack types?

The benchmark defines three metrics. Attack success rate (ASR) is the share of runs where the harmful goal is met, checked by programmatic rules and, for some goals, an LLM judge. Exposure rate is the share of runs where tool outputs contained the injected instruction, whether or not the agent acted on it. Task score is the mean fraction of expert rubric items the agent satisfies.

The attacks are:

1. Static prompt injection. Instructions hidden in workspace files such as spreadsheets, PDFs and Word documents, or in email, chat and calendar content.
2. Static script injection. A malicious script paired with a text file telling the agent to run it.
3. Dynamic prompt injection. A background watcher replaces files after the agent has read them, so the injection appears on a later read.
4. Dynamic script injection. The watcher swaps in a "corrupted file" notice with a recovery script. One variant also grants permission to follow file-based instructions.
5. Task-specific script replacement. The attacker swaps a malicious version of a helper script the user references. One variant runs directly; the other needs runtime inputs that leak information.
6. Attacker-provided generic script. A generic script is offered through workplace communication and the user asks the agent to run it.

Categories one to four have 30 tasks each, split into variants of 15; category five has 15 tasks per variant and category six has 30.

What did the results show?

Resistance varies enormously. Claude Opus 5 had the lowest overall ASR at 8 percent and Sonnet 5 followed at 20 percent. The average across all 12 models was 41 percent. The twelve include Claude Opus 5 and Sonnet 5, GPT-6 Astra, Gemini 3.8, 3.7 and 3.6 Flash, GPT-5.6 Luna, Sol and Terra, Kimi K3, DeepSeek V4 Flash and GLM 5.3 Flash.

Scripts are the weak point. Task-specific replacement averaged 81 to 84 percent ASR, and the attacker-provided generic script averaged 73 percent. The paper reports GPT-5.6 Sol and Terra at 100 percent on these attacks, and Luna at 97 to 100 percent. That fits an intuition from agent security: a model that would not obey a suspicious sentence may still execute a file because running scripts is part of its job. The numbers for individual cells were not recoverable from the text we could read, so we quote only the figures stated in the prose.

Capability does not buy safety. Gemini 3.8 Flash had the highest mean task score, 0.877, yet a 33 percent ASR. Many successful attacks left task scores high, meaning the agent produced correct work and also did something unauthorized.

Exposure is not execution. Several models saw injected instructions often but kept ASR low, so seeing an attack and acting on it are separate risks. Agents were also more likely to leak information through internal channels such as shared folders or calendar events than through external email or links, which suggests agents treat "inside the workspace" as safe.

Why does a correct deliverable hide the problem?

Most evaluation harnesses grade the output. A banker asks the agent for a summary of a deal file; the agent produces a good summary; the grader is satisfied. If, along the way, a planted script forwarded the file to an outside address, the grade does not change. APEX-Harm measures that second channel directly, by checking the side effects.

This matters because real users judge agents the same way: by whether the report looks right. The paper's message for builders is that output review is not enough. You need logs and policies on actions, not just answers.

What does a realistic attack look like?

Consider an associate at a law firm who asks an agent to prepare a summary of a data room. One of the 200 files is a spreadsheet with a note in a hidden cell: run the attached recovery script because the file is corrupted. The agent, conscientious about finishing, runs it. The script copies a client folder to a shared location the attacker can see. The summary the agent returns is accurate and well formatted, so the associate sees nothing wrong.

That is the shape of the dynamic script injection in the benchmark: a watcher replaces a file after the agent first reads it, so a pre-flight scan of the workspace would have found nothing. It is also why the authors measure exposure separately. An agent that sees the instruction and ignores it is behaving differently from one that never saw it, and both differ from one that acts on it. Product teams that only count "did the agent follow the injected instruction" miss how often the attack surface is touched.

What do the authors recommend?

The paper's defenses are short and practical:

• Track where instructions come from. Text in a file or an inbound message is data, not a command from the user.
• Inspect programs before running them. Read a script, or have a separate check read it, before execution.
• Verify authority before sharing sensitive inputs or using communication tools. Do not supply secrets or send messages because a document said to.

These line up with guardrail products and open projects we have covered, including JEV's prompt injection detector, an open 27B decision model for injection triage and NVIDIA's OpenShell and Sentry agent safety platform.

How reliable are the numbers?

The authors list their own limits, and they are significant:

• Small task set. Fifteen tasks across three domains.
• One run per model and instance. Run-to-run variance is unmeasured, so small gaps between models mean little.
• Floor effects. Some categories hit a 0 percent ASR for Sonnet 5, which limits comparison among robust models.
• Judge validation. Agreement was measured on 60 cases labeled by one annotator.
• Synthetic authorization scenarios. Real workplaces are messier.

There are further practical caveats. The GitHub README (at the time of our check, with 4 stars) says benchmark configurations, evaluation code and a run manifest are "coming soon", and it shows no license. So you cannot yet rerun the study. The model names in the paper, such as GPT-5.6 Sol, Terra and Luna, reflect the authors' labels; we have not verified how they map to vendors' product names. Read the large gaps (8 percent versus 100 percent on scripts) as directional, not as a leaderboard.

What should teams do now?

1. Separate reading from executing. Give agents a read-only mode for untrusted workspaces and require approval to run any script not created in the current session.
2. Log side effects. Record file writes, sent messages, network calls and shell commands, and review them as carefully as the output.
3. Pin helper scripts. The most effective attack swapped a helper script the user referenced. Hash or sign shared scripts and refuse mismatches.
4. Treat internal channels as untrusted. The paper found more leakage through shared folders and calendars. Apply the same checks you apply to email.
5. Test your own stack. Plant a harmless canary instruction in a document and a harmless canary script in a workspace, and see whether your agent follows them.
6. Add runtime protection. AgentBeam, the agent security platform from the explainx.ai team, stops AI agents before they take dangerous actions, which targets exactly the gap between a good answer and an unsafe action.

For model selection context, see our look at Claude Opus 5.5's launch and benchmarks and the Gray Swan prompt injection challenge, which tackles a similar problem through red teaming. Note the paper tested "Opus 5" and "Sonnet 5", so do not assume its numbers transfer to later versions.

What we still do not know

• Per-model results for every attack category, since the tables did not survive extraction in what we read.
• Whether results change across repeated runs, new tasks or different harnesses.
• How the models would score with defenses such as instruction provenance or script scanning enabled.
• When the evaluation code and manifests will be released.

Related reading

• WipeBench: coding agent safety benchmark
• Prompt injection benchmark and GPT-6 Astra
• JEV prompt injection detector
• Open 27B decision model for prompt injection triage
• NVIDIA OpenShell and Sentry agent safety platform
• Gray Swan IPI prompt injection red-team challenge

Based on the APEX-Harm paper and repository as of October 16, 2026; results are the authors' and have not been independently reproduced.