Multimodal Knowledge Agent (MKA): Google's Benchmark for Visual Agent Work

Google and Artificial Analysis launched the Multimodal Knowledge Agent (MKA) benchmark on November 10, 2026, the same day Gemini 4 Carbon went broadly available. The suite measures how well agents reason over visually complex file bundles when producing real knowledge-work artifacts, not multiple-choice answers scraped from plain text.

If you tune agent stacks for finance decks, construction drawings, or lab notebooks, MKA is the first Google-backed suite that treats layout perception as a first-class scoring dimension.

TL;DR

Why Google built MKA now

Text-heavy leaderboards plateaued in 2025 and 2026. Teams could win finance Q&A or doc summarization suites without reliably reading charts, stamping coordinates on blueprints, or checking slide typography. Yet that is how analysts, architects, and underwriters actually work.

Google''s Introducing the Multimodal Knowledge Agent benchmark post argues the gap explicitly: most knowledge benchmarks never force the model to open an image. MKA instead ships multi-file prompts where success means a polished deliverable — spreadsheet, brief, deck, or CAD-derived model — judged against expert rubrics.

The timing lines up with Google Antigravity demos that build Three.js exhibits from PDF engineering plates and Carbon''s /swarm preview. Google wants a number that captures perception + long horizon together, not only cyber exploit games.

Three cream grippers place different green shapes into one shallow round tray

Task design: three to eight hours of professional work

Each MKA task mirrors work a specialist would bill three to eight hours for:

1. Realistic natural-language prompt describing outcomes, not tool calls.
2. Multimodal input bundle (PDFs, images, CSV extracts, slides).
3. Golden reference artifact for rubric authorship (not leaked to models during private evals).
4. ~70 binary rubric criteria weighting factual accuracy (75%) and visual design (25%).

Domain experts across 83 occupations co-authored tasks. Google lists verticals spanning construction, healthcare, logistics, and software/data, reflecting jobs where misreading a legend or gridline breaks the downstream spreadsheet.

Contrast with WorkflowArena''s orchestration focus: that benchmark scores who delegates whom. MKA scores whether the deliverable looks like a professional finished it, down to chart labels and slide hierarchy.

Contrast with SEC-bench and ExploitGym: those suites reward offensive cyber moves. MKA rewards getting the commission numbers right inside a branded template.

Stirrup harness and the private 200-task lock

Artificial Analysis runs every entrant in Stirrup, an open harness with fixed tools, sandbox, and turn budget. Google''s post highlights two anti-cheat choices:

• Private 200-task set so tasks cannot leak into training corpora or be solved via search snippets.
• Public 10-task set on Hugging Face so researchers can inspect JSON schemas and smoke-test local agents.

Grading uses an agentic autorater built on a non-Google model. It opens each deliverable, renders pages, inspects spreadsheet formulas, and applies the rubric. Google reports 96% agreement between autorater and human experts when they double-scored outputs.

For builders, that means your CI should log intermediate files, not only final chat transcripts. Raindrop-style memory harnesses and semantic caching still help, but MKA punishes pretty summaries that fail cell H14.

Efficiency spreads: 30 to 130 turns on the same task

Google''s MKA announcement notes wide turn-count spread on identical tasks: some frontier models burn 30 turns, others 130, before submitting. That matters for token economics when you price agent fleets per seat.

The report also ablates vision: models try compensating with text-only hacks and fail rubric checks that require spatial alignment. If you are compressing screenshots to tiny thumbnails to save money, MKA is basically a stress test for that shortcut.

Gemini 4 Carbon on the quality–cost frontier

Google ties MKA scores directly to Carbon''s go-wide moment:

• #2 quality among tested models.
• ~6× lower average cost per task than the highest-quality model in Google''s published chart.

Treat those as vendor-reported Pareto points until Artificial Analysis publishes interactive tables. Still, the claim matches Google''s broader narrative that Carbon buys long verifiers more cheaply than rivals — the same story as CursorBench and AutomationBench wins in Koray Kavukcuoglu''s availability post.

Teams comparing frontier models should add one MKA-style internal rubric (even five tasks) before routing Antigravity /swarm at production customer PDFs.

How to start evaluating your own stack

1. Pull the 10 public Hugging Face tasks and read prompt + rubric JSON.
2. Run your agent in Stirrup (or replicate its tool surface) with identical turn caps.
3. Log perception steps separately from reasoning steps so you can see when the model skips rendering.
4. Autograde with a second model plus human spot checks on high-stakes rubric lines (legal numbers, dose calculations).
5. Compare cost per successful rubric point, not raw pass rate — MKA''s economics mirror FinOps tokenomics thinking.

If you lack GPU budget to run Carbon locally, API evals through Gemini for business are enough for relative harness tuning, even when absolute scores shift.

Public Hugging Face tasks: how to read the format

Google and Artificial Analysis shipped 10 public tasks so you can study schemas before paying for hosted evals. A typical task bundles:

• Prompt prose describing stakeholder deliverables (for example, "Produce a variance analysis deck comparing Q1–Q3 site photos against the permit PDF").
• File manifest listing PDFs, PNGs, CSV extracts, and CAD exports with checksums.
• Rubric JSON where each line is a yes/no question with weight tags (accuracy vs design).
• Reference artifact metadata explaining what a perfect submission looks like without leaking hidden test answers into your training pipeline.

When you port tasks into your internal CI, do not paste rubric text into model prompts. That recreates the leakage MKA tries to avoid. Instead, keep rubrics in your grader process exactly like Stirrup does.

Walk through one task with your design team. Architects will immediately spot when the rubric checks dimension strings on drawings, while generic LLM judges ignore millimeter tolerances. That gap explains why Google reports large turn-count spread: models that skip rendering burn turns guessing.

Stirrup vs your bespoke harness

If you already built a custom LangGraph or Claude Code plugin, map components to Stirrup equivalents before comparing scores:

Aligning surfaces lets you reuse public HF tasks even when you are not running Stirrup binaries yet.

Carbon vs Argon routing on MKA-style work

Google positions Argon for mainstream engineering and Carbon for frontier perception + verification (availability post). On visually dense rubrics, Argon may finish faster while Carbon buys accuracy on chart legends and CAD overlays.

Run A/B harness tests with identical Stirrup caps:

1. Argon-only baseline for cost.
2. Carbon planner + Argon workers via /swarm if you are on Antigravity Ultra.
3. Carbon-only for regulated deliverables where misses are expensive.

Log dollars per rubric point using the same tokenomics math you use for Bedrock, even when you pay Google directly.

Building an internal five-task smoke suite

You do not need all 200 private tasks to benefit. Copy the rubric structure into five internal documents your company already uses (sales proposals, SOC2 evidence packets, PCB reviews, insurance binders, lab slide decks). For each:

1. Strip customer identifiers.
2. Write 30–50 rubric checks with weights.
3. Run your production agent weekly.
4. Track regressions when you upgrade models.

This is how Dawes-style peer review benchmarks started — small planted-error sets before public leaderboards.

Limits builders should remember

• Rubric autorater bias: Any autorater can over-weight formatting vs substance. Keep a human review queue for regulated outputs.
• Private set opacity: You cannot reproduce leaderboard numbers without Artificial Analysis'' hosted run.
• English-centric occupations: Expert pools may not cover every locale''s document norms.
• Google co-authorship: Carbon headlines are marketing plus science; cross-check on independent suites when they add MKA-derived tasks.

Grading disagreements: what 96% agreement leaves out

Four percent disagreement between autorater and experts is enough to flip leaderboard ranks when tasks are hard. Run dual grading on your top five business-critical templates before trusting any vendor autorater blindly.

When experts disagree with automation, capture rationale strings — they become training data for the next rubric revision.

Artificial Analysis hosted runs

MKA leaderboards live on Artificial Analysis infrastructure with Stirrup locked. That is good for fairness, bad for air-gapped enterprises. If you cannot upload files externally, replicate rubric structure locally and treat public numbers as directional.

Link eval cadence to model release trains — especially Carbon pricing changes Nov 30.

A flock of dots flowing in a wave with a few green leaders

FAQ deep dive for platform leads

Should we block Carbon until MKA replicates locally? No — use Google''s public tasks plus five internal rubrics. Block only if your compliance regime forbids closed private sets entirely.

Does Stirrup lock us into Google tools? The harness is open source; tools inside can call any HTTP API your security team allows.

Will MKA replace document OCR benchmarks? Unlikely. OCR suites measure character error rate; MKA measures downstream professional deliverables. Use both.

Related reading

• Gemini 4 Carbon launch and November 10 availability
• Vertex access claimed vs verified same day
• WorkflowArena: orchestration benchmark
• Frontier model comparison table
• Multimodal vision token budgeting
• Google Antigravity /boost deep reasoning