An open benchmark from Our Secure Future reports that frontier AI models score about 0.82 on Women, Peace and Security (WPS) analysis when a prompt cues it, and about 0.16 when it does not. The authors call this the "WPS Competence Gap." The finding is in a guest essay on the Collective Intelligence Project's Substack by Moira Whelan and Jesper Frant, dated October 21, 2026, which reached Hacker News on November 12.
The point extends past one UN mandate. It asks a general question about AI assistants: do they raise the considerations a competent expert would flag, or only the ones the user asks about? This post explains the method, the numbers, the limits the authors state, and how builders can use the data.
TL;DR
| Item | Detail |
|---|---|
| Who | Our Secure Future, a program of PAX sapiens, with the essay by Moira Whelan and Jesper Frant |
| What | Benchmark of 30 scenarios and 11 criteria on Women, Peace and Security |
| Headline | Frontier-model average 0.82 cued, 0.16 on sparse prompts |
| Best score | Multi-agent workflow on GPT-5.6 Sol, 0.84 overall |
| Caveat | Not a fitness-for-deployment certification; only three adversarial scenarios |
| License | CC BY 4.0, free, usable as training data |
What the benchmark tests
WPS is a UN Security Council mandate, beginning with Resolution 1325, that asks security institutions to consider how conflict affects women and to include women in decisions about peace and security. The authors picked it because militaries, ministries and UN missions are starting to use language models to draft situation reports, engagement plans and conflict analyses, as one author explained in the Hacker News thread. The question is whether a model brings up WPS on its own.
They describe two failure modes:
- Passive omission. Models leave gender out of routine analysis unless asked.
- Active capitulation. Models help justify dropping WPS structures when budget or operational pressure is applied.
Method
According to the essay, the benchmark has:
- 30 scenarios in five tiers: context-rich (8), cue-stripped pairs (4), gender-neutral (7), sparse or ambiguous (8) and adversarial (3).
- An 11-criterion rubric including Gender/WPS Integration, Policy Alignment, WPS Guardrails and Substantive Pushback, plus analytical depth and factual integrity.
- Three LLM judges: gemini-2.5-flash, gpt-4.1-mini and claude-haiku-4.5, using Weval's method. The overall score is the equally weighted mean of the five tier means.
- Practitioner review. In June and July, experts gave over 140 comments. Their feedback led to the adversarial and cue-stripped tiers and to a definition of gender as socially constructed norms.
- A prototype corpus of 215 National Action Plans and 141 research documents that informed an early WPS agent.
The authors tested Gemini 3.1 Pro and Preview, GPT-5.6 Sol, Claude Opus 5, Gemini 2.5 Flash and GPT-4o. The WPS AI Leaderboard says scores run from 0 to 1, with 0.60 as the pass threshold and 0.75 and above labeled "strong," and that differences under roughly 0.10 to 0.15 should be read as directional.
What they found
The blind spot. On sparse prompts, the frontier-model average fell from 0.82 with cues to 0.16, per the authors, and the blind spot was uniform across models. The bare GPT-5.6 Sol model dropped from 0.86 cued to 0.21 sparse.
A concrete example. For an intelligence summary about displaced people and a food price spike, the authors say Gemini 3.1 Pro and GPT-5.6 Sol never mentioned women, girls or gender, and left sexual violence unaddressed. Gemini also proposed using displaced people as an intelligence source.
Capitulation under pressure. Given a 15% budget cut to gender roles, the authors say Gemini 3.1 Pro Preview recommended eliminating the gender advisor post and drafted a justification for leadership. Claude Opus 5 and Gemini 2.5 Flash pushed back, citing mandate non-compliance, funding clawback risks and operational returns. The authors note that resistance did not track model size or general capability.
What helped. Prompting and retrieval helped but depend on users already asking gendered questions. A multi-agent workflow on GPT-5.6 Sol, with intake, specialist agents, synthesis and a verification pass, scored 0.84 overall, the top score on the leaderboard, with gender sensitivity of 0.87 on both cued and sparse prompts. GPT-4o with the workflow moved to second place but still showed a drop between tiers. The authors call scaffolding "a Band-Aid" and want labs to address gender in training.
How to read these numbers
A benchmark like this is easy to over-read. Several limits apply, and the authors name some of them.
- Judges are models. All three judges are LLMs. The leaderboard says judge-agreement caveats are on its methodology page, and small gaps are directional only.
- Small adversarial set. The authors say three adversarial scenarios are too few for firm conclusions. The Gemini 3.1 Pro Preview capitulation example is one scenario.
- Rubric rewards raising the topic. The test measures whether a model volunteers WPS content. Whether a human expert would always want that content in every context is a judgment, which is why the authors involved practitioners.
- Not a certification. The authors say it is not a deployment certification and that human WPS advisors and direct consultation with affected women and communities remain essential.
- Dated models. The scenarios are valid only for models released before the benchmark, and the model list includes older systems such as GPT-4o. New releases may behave differently.
- Authors have a service. Our Secure Future offers custom evaluations, procurement advice and team training for a fee. That does not make the findings wrong, but it is context.
The leaderboard page we fetched showed unfilled template placeholders in place of rows, so we could not check the live ranking ourselves. All scores here are the authors' reports from the essay.
Why it matters to builders
The pattern generalizes. Any assistant used for analysis can omit a domain consideration unless the prompt names it. That affects legal review, medical triage, safety cases and security planning as much as peacekeeping. The same shape appears in other evaluations: a model "knows" a fact when asked and misses it when it is not. We saw a version of that when Vals AI tested language-dependent leanings in 25 models, where the answer depended on how the question was framed.
Practical steps for teams that build analysis tools:
- Test with sparse prompts, not just cued ones. If your eval always names the issue, you measure recall and miss omission.
- Add a verification pass that checks a draft against a domain checklist. The authors found scaffolding raised scores, with the caveat that it is a patch.
- Test pressure scenarios. Ask the model to justify cutting a safeguard and see if it pushes back.
- Use more than one judge model and treat small score gaps as noise.
- Keep a human expert in the loop for high-stakes work.
The policy angle
The authors recommend that large procurers such as NATO and the UN evaluate AI systems against WPS benchmarks, that labs build gender dynamics into training, that organizations write WPS requirements into AI procurement, and that evaluations test positive behaviors defined by domain experts, not only red lines. That last point links to wider safety work. OpenAI's pledge on children and AI safety research and the Paris Call on AI safety and fail-safe shutdowns show institutions asking for domain-specific checks. Government use of AI in security work is growing too, as in our post on OpenAI national security staff and a prank call.
What was said on Hacker News
The thread had about 20 points and 21 comments when we checked. One author, frantj, described the project and its aim. Commenters added context for readers who did not know the mandate, and some questioned the premise or the length of the essay. We found no hands-on replication in the thread, so none of the scores have independent confirmation yet.
Cued, sparse and adversarial: what the tiers mean
The tier names carry most of the meaning, so here is a plain reading. A cued prompt points at the issue, for example by mentioning women or gender in the scenario. A sparse prompt describes the same kind of situation with little detail and no cue, so any WPS content must come from the model. A gender-neutral prompt avoids the topic on purpose, and a cue-stripped pair is a cued prompt rewritten without the cue, which lets the authors compare the same scenario with and without the hint. Adversarial prompts add pressure, such as a budget cut, to see whether the model argues for keeping a safeguard or helps remove it. The 0.82 to 0.16 gap compares the first and the sparse group, so it measures what happens when the hint disappears.
How to try it
The authors say the benchmark is free under CC BY 4.0 with permission to use it as training data, and that you can run it on Weval to reproduce the leaderboard. The leaderboard site lists version 6 as a downloadable zip and a 30-row dataset on Hugging Face with a README and reliability document. If you build an assistant for analysts, run it against a sample of the sparse scenarios and compare your results with the reported 0.16 baseline.
Related reading
- Vals AI language-dependent leanings in 25 models
- OpenAI pledges $2 million for children and AI safety research
- Paris Call on AI safety and fail-safe shutdowns
- OpenAI national security staff and the prank call
- Anthropic's US midterms safeguards for Claude
- Gemini 4 Hydrogen vs Argon and Carbon
This post reflects sources available on November 12, 2026. Scores are the authors' reports.