Most advice for making a vision model smarter is "let it think longer." A new paper from UCLA NLP argues that is the wrong lever. OpenVLHarness, released this week, wraps an existing multimodal model in a set of visual tools and a persistent memory, and reports average gains of 9.7 to 12.6 points across 22 benchmarks. No weights are changed.
The paper (arXiv 2610.26860) was submitted on October 19, 2026 and went viral on X after AK shared the demo video on October 20. This guide covers what it does, how to run it, and which numbers to trust. If you are new to the term, start with our explainer on what an agent harness is.
TL;DR: questions builders ask first
| Question | Answer |
|---|---|
| What does it do? | Gives a multimodal LLM visual tools plus memory that keeps tool outputs as reusable artifacts |
| Do I need to fine-tune? | No. The README says "nothing is fine-tuned" and it works through native tool calling |
| Which models? | Qwen3-VL Instruct, GPT-5 to GPT-6.1 (Luna, Sol, Astra) and Kimi K3 presets |
| License? | MIT for the code; third-party model and dataset licenses apply |
| Cost? | The paper says at most 37% of the model-token cost of the compared systems |
| Needs extra services? | Yes. Tool servers for SAM 3, OCR and depth, plus a Serper key for search |
| Sandboxed? | No. The Python tool runs on your host; isolate it yourself |
| Maturity? | New. About 121 stars, 3 forks and 1 open issue when we checked |
What problem does OpenVLHarness solve?
Many visual questions need evidence a model cannot produce by reasoning alone: counting 300 small objects, reading tiny print, judging distance from a photo, or checking a fact against the web. The authors' controlled study across 22 benchmarks found that "simply increasing the inference budget of the base model yields only modest gains."
Their answer is a harness: a loop around the model that decides when to look closer, measure, search, or compute. The core claim is that capability should live in the layer around the model, so the same layer can lift both open and proprietary backbones. That is the same argument made in our posts on agent harness engineering and why the harness is the company, applied to vision.
How does it work?
The architecture has three parts, per the project page and README.
Rounded agent figure wearing a belt of tools, illustrating how OpenVLHarness gives a multimodal model visual tools
1. The orchestrator. Your multimodal LLM plans, calls tools across steps and writes the code that the coding tool executes.
2. The capability layer. This turns heterogeneous tool outputs into observations the model can read.
| Capability | Default backend |
|---|---|
| Visual grounding | SAM 3 |
| OCR | PaddleOCR-VL |
| Depth estimation | Depth Anything 3 |
| Camera trajectory | Depth Anything 3 |
| Zoom-in | Built-in image processing |
| Text search | Serper plus Qwen |
| Image search | Serper Lens plus Qwen |
| Webpage visit | Serper scrape plus Qwen |
| Python coding agent | Orchestrator backbone plus local Python |
3. Persistent multimodal memory. Images, boxes, masks, depth maps and web references stay addressable for the whole question. Later tools and code can reuse them. Each new question starts with fresh memory, and the coding agent runs in a separate context so it can compute over stored results without bloating the main conversation.
One reply on X put the practical point well: keeping tool outputs as addressable artifacts instead of pasting them back into context may explain why a medium-effort model plus harness can beat a max-effort model alone. That is a reader's guess, not something the paper tests, but it fits the design. Our guide to multimodal AI covers the basics if the vocabulary is new.
Do the results hold up?
The README reports 22-benchmark averages. Benchmarks span counting and grounding, multimodal search, general VQA and hallucination, and spatial understanding.
| Backbone | Base | With OpenVLHarness | Gain |
|---|---|---|---|
| Qwen3-VL-8B-Instruct | 34.24 | 45.41 | +11.17 |
| Qwen3-VL-32B-Instruct | 38.30 | 50.92 | +12.62 |
| Kimi K3 (max effort) | 56.43 | 66.09 | +9.66 |
| GPT-5 (medium) | 44.37 | 55.65 | +11.28 |
| GPT-6 Luna (medium) | 45.76 | 56.52 | +10.76 |
| GPT-6 Sol (medium) | 52.67 | 64.31 | +11.64 |
| GPT-6 Astra (medium) | 60.27 | 70.17 | +9.90 |
Two patterns stand out. First, the gain is similar for a free 8B model and a frontier proprietary model, which suggests the harness is adding something the models lack, not just compensating for weakness. Second, the strongest model gains the least, as you would expect near a ceiling. The project page adds that with GPT-6 Astra all 22 benchmarks improve.
Harness comparisons
On GPT-6 Luna, the project page lists Visual Sketchpad at 45.6, Codex at 43.8 and OpenVLHarness at 56.5, against a base of 45.8. So the other harnesses did not beat the plain model on this average. On a Qwen2.5-VL-7B backbone, OpenVLHarness with original weights scores 33.5, against fine-tuned tool-use systems at 25.0 to 28.0 (AdaReasoner, PixelReasoner, DeepEyesV2).
For comparison, our roundup of the top open and closed source agent harnesses covers coding-focused harnesses; OpenVLHarness is a rare vision-first entry.
What should you be careful about?
Read the claims with these caveats.
- The viral numbers mix comparisons. The X thread says the harness "beats Codex and other VL harnesses by 7.7 to 8.5 points, costing 37% as much or less." The abstract frames that range as GPT-6 Luna at medium effort against the base model, Codex and Visual Sketchpad at maximum effort. The project page's matched-effort table shows larger gaps of roughly 10.7 to 12.7 points. We read the 7.7 to 8.5 range as the max-effort comparison, but we could not confirm that from the page alone.
- The cost claim has no public numbers. The abstract says at most 37% of the compared systems' model-token cost. The project page shows a cost chart but gives no figures we could read, so the ratio is unverified here.
- Run counts differ. Qwen3-VL results are means of three runs. GPT-6 and Kimi K3 results are single runs, per the project page.
- It is a preprint. v1 was posted on October 19, 2026, and we are not aware of peer review.
- Maturity. The repository showed about 121 stars, 3 forks and 1 open issue. Expect rough edges.
- Search tools need extra setup. A Serper API key is required, and reverse image search also needs image hosting such as a tunnel, S3 or GCS.
How do I run it?
Python 3.10 or newer is required. From the README:
git clone https://github.com/uclanlp/OpenVLHarness.git
cd OpenVLHarness
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[qwen,server]'
Then export the connection variables (MODEL_BASE_URL, AUXILIARY_MODEL_BASE_URL, JUDGE_MODEL, TOOL_SERVER_URL and the SAM 3, OCR and depth server URLs). Start the router and check it:
python -m openvlharness.services.server --config configs/agent/qwen3_8b_paper.yaml
curl --fail http://127.0.0.1:8101/health
From Python, call build_agent(...) and agent.generate([...]); the result includes a prediction and the trajectory. To run the benchmarks:
openvlharness-eval \
--config configs/agent/qwen3_8b_paper.yaml \
--judge-config configs/judge_no_judge.yaml \
--data FSC147_val ODinW13_bbox_val_qwen3 \
--num-examples 12 --shuffle --seed 42 \
--num-proc 1 --save-html all --work-dir /path/to/local/runs
Every preset uses Qwen3-VL-8B-Instruct as the auxiliary model for search summarization and webpage extraction, so even a GPT-6 setup needs that model reachable. A hosted demo on Hugging Face lets you try it with your own OpenAI-compatible key before installing.
Tip: capabilities can be turned off in the YAML. examples/local_tools.yaml uses only zoom and the coding tool, which removes most of the service sprawl for a first test.
Is it safe to run on my machine?
Not without isolation. The README is blunt: "Its execution timeout is not a sandbox; run the agent in an isolated environment appropriate for your data." The coding tool executes model-written Python on the agent host with your permissions, including network and filesystem access.
Treat it like any code-executing agent: use a container or disposable VM, mount only the data it needs, and keep credentials out of the environment. If you run agents that execute code and want a policy layer that stops dangerous actions before they happen, AgentBeam, from the explainx.ai team, is built for that.
Where did the harness come from? Automated optimization
The project page describes a second idea worth noting. Kimi K3 critiqued 9,893 failed Qwen3-VL-8B trajectories and assigned each a root cause and a remedy channel: a new task-specific tool, a change to harness text, or neither. A coding agent then edited system prompts, tool descriptions and output templates, and composed new tools from existing ones, all with model weights fixed.
That is the same pattern we covered in Google's Harness2 recursive harnessing work: let a stronger model tune the harness of a weaker one. It is also close in spirit to Microsoft's Agent Lightning, though that work uses reinforcement learning.
What this means for what you build
- Try the harness before the bigger model. If your product reads photos, receipts, charts or screens, a tool layer may beat paying for a larger backbone or more reasoning effort.
- Copy the memory pattern. Store tool outputs as named artifacts and reference them by ID, instead of re-pasting images and crops into context. This applies to any agent, not only vision ones.
- Plan for ops cost. Free weights do not mean free serving. You will run or rent SAM 3, OCR and depth services, and pay for search.
- Isolate code execution. The pattern that makes it powerful, a model writing Python over stored data, is also the risk.
For the computer-use side of visual agents, see how Codex computer use controls Windows and mobile.
Claimed versus verified
| Claim | Status |
|---|---|
| Average gain of 9.7 to 12.6 points over 22 benchmarks | In the abstract and README; not independently reproduced |
| GPT-6 Astra +9.9 (60.27 to 70.17) | README table |
| Beats Codex and Visual Sketchpad on Luna | Project page table; baselines scored below the plain base model |
| 7.7 to 8.5 point margin | Appears to be a max-effort comparison; unconfirmed |
| At most 37% of the token cost | From the abstract; no public figures found |
| Training-free, MIT licensed | README |
| Production ready | Not claimed; new repo, no sandbox |
Related reading on explainx.ai
- What is an agent harness? Complete guide
- Top 10 open and closed source agent harnesses
- Agent harness engineering and Terminal-Bench
- The harness is the company
- Google Harness2: recursive agent harnessing
- What is multimodal AI?
- Codex computer use on Windows and mobile
- Official: paper, code, project page
Version details, star counts and benchmark numbers are accurate as of October 21, 2026 and may change as the repository and paper are updated.