Property Finder Found 96.8% of Its Eval Labels Came From the Model Being Tested

If you ever compared a new model against your production model using the production model's own outputs as the answer key, you have measured similarity, not accuracy. An AWS Machine Learning Blog post published October 27, 2026 describes exactly that failure at Property Finder, a MENA real estate platform: 96.8 percent of its evaluation labels came from the incumbent model. This post walks through the case study, the method and the limits AWS itself states.

It pairs well with our write-up of the Exowatt fine-tuning case, published the same day, where the lesson is also about measuring honestly.

Quick reference

How a biased answer key happens

Property Finder checks every listing image for watermarks, room type, furnishing status, photo quality, construction indicators and more, with about 15 images per listing. Wanting a cheaper model, the team built an evaluation set the common way: run the current model and treat its outputs as truth. It was running in production and nobody complained, so the labels seemed fine.

The result was circular. Any alternative was judged by how closely it matched the incumbent, not by whether it was right. Hidden errors confirmed it: watermark text extraction had character-level mistakes marked correct, including hallucinated brand names; photo classification forced single labels onto images with several rooms; and in the 3.2 percent of human-labeled samples, annotators had added categories outside the specification.

Building ground truth you can trust

• Independent experts. A strategy and analytics expert with real estate listing knowledge annotated all images, with no access to the incumbent's predictions. The AWS Generative AI Innovation Center ran clarification sessions on ambiguous cases.
• Dataset scope. 356 UAE images. The first 256 formed the development set; all 356 form the final evaluation set. AWS says about 356 images is roughly enough to detect moderate differences and that rare classes may be underpowered. A separate 230-image Egypt set served as a cross-regional check and is not reported.
• Multi-label output. A hallway photo showing a bedroom through a doorway is genuinely both, so allowing multiple labels measures real correctness rather than penalizing reasonable secondary predictions.
• Metrics. Micro F1 as primary, Macro F1 alongside because Micro is dominated by frequent classes, plus per-label precision and recall, per-sample intersection over union, and exact-match accuracy on nine single-label attributes.

AWS also suggests a hybrid for future expansion: models from different families propose labels and disagreements, while humans adjudicate every case so no model output becomes a label directly.

Prompt optimization as an evolutionary loop

With honest labels, the team improved prompts through 27 versions. AWS frames it like reinforcement learning: each prompt version is an agent, Micro F1 is the reward and error analysis supplies the direction. It resembles GEPA, reflective prompt evolution, and later versions used MIPROv2, an automated optimizer from the DSPy library.

Concrete fixes from error analysis show what the loop looks like in practice:

• Rooms seen through doorways were being labeled as secondary rooms; an instruction to classify by the room occupying most of the frame came first, then multi-label output.
• Decorative Arabic text overlays in some regions were not watermarks; rules for them removed a class of false positives.
• Prompt v21, multi-label, became the shared starting point. Llama 4 Maverick 17B Instruct scored 0.75 Micro F1 on it, then 0.81 with v27 on the development set.
• The final v27_PF added rules for renders, bordered images and corridors.

Micro F1, Macro F1 and why both are reported

Micro F1 pools every prediction across every label before computing precision and recall, so common labels dominate it. That matches a business goal of overall label correctness, which is why Property Finder used it as the headline. Macro F1 computes F1 per label and averages them, so a rare class such as an unusual room type counts as much as a common one. AWS reports Macro alongside Micro because a model can look strong on Micro while failing on rare attributes.

The per-sample intersection over union adds a third view: for each image, how much the predicted label set overlaps the expert's. It is the natural metric once you allow multiple labels per image, because a prediction of bedroom and hallway for an image the expert labeled bedroom, hallway and corridor is partly right rather than simply wrong.

The practical advice is to pick the metric that matches the decision. If every image is checked once and an error costs about the same, Micro F1 works. If missing a rare but costly case is the real risk, watch Macro F1 and per-label recall.

Why the incumbent looks better than it is

There is a subtle statistical reason the circular labels were so damaging. A model's mistakes are correlated with itself: if it systematically calls corridors bedrooms, a label set copied from its outputs rewards any model that makes the same mistake and punishes any model that gets corridors right. The challenger is therefore penalized precisely where it is better. Property Finder's own data showed it: alternatives looked worse than the incumbent until fresh expert labels revealed that the incumbent's labels contained hallucinated watermark text and forced single-room labels.

This is not unique to vision. Teams comparing coding agents, summarizers or classifiers often seed a benchmark from the current system's logs. The safe pattern is the one AWS describes: label independently, keep the test split sealed, and treat any agreement with the old model as a signal to investigate, not a score.

What failed

AWS lists failures, which is useful. Splitting watermark attributes into a separate API call gave no meaningful accuracy gain and hurt boxed-image detection, because models mistook watermark overlays for borders. Generic prompts failed on every model tested, since real estate imagery is too domain-specific for one-size-fits-all instructions.

Results and how to read them

With v27_PF, Llama 4 Maverick 17B Instruct scored 0.80 Micro F1 on the 356-image evaluation set against 0.56 for proprietary model A on its v22 prompt. Exact-match accuracy on single-label attributes: rotation detection 100 percent, people 97, boxed image 97, watermark detection 94, render 93, furnished status 91, room under construction 81, room size 65 and watermark text extraction 55. It matched or beat model A on all 11 attributes; on photo classification they tied.

Read these carefully, using AWS's own caveats:

• The evaluation set includes the 256 development images, and both prompts were tuned on them, so the scores are optimistic.
• Model A got one tuning round; Llama got several (v24 to v27) plus Property Finder's edge-case rules, so it had more effort behind it.
• The cost figure is an estimate as of December 2025 in US East (Ohio), assuming 9,000 input and 200 output tokens per image; the production prompt uses about 7,500 to 8,000 input tokens, so it is an upper estimate. The 86 percent compares against the earlier setup's estimated cost using v21.
• The 40 to 60 percent latency drop is Property Finder's own monitoring, not a published benchmark.

A checklist for your own model migration

1. Trace every label to its source. Count how many came from a model.
2. Label a few hundred samples independently. Domain experts, no model outputs visible.
3. Hold out a split you never tune on. Property Finder's reuse of the dev images is the weakness AWS itself flags.
4. Choose metrics that match the business. Micro F1 for overall correctness, Macro F1 for rare classes.
5. Tune each candidate's prompt separately and equally. Unequal effort confounds the comparison.
6. Compute cost per 1,000 items from measured tokens. Re-check with current prices.
7. Use a disagreement signal for human review. AWS suggests two models disagreeing as a practical flag, and recommends checking how often it catches real errors first.

Why this matters

The comparison between models is only as good as the answer key. Many teams migrating between providers in 2026, a common move as prices and capabilities shift, will inherit labels produced by the system they are replacing. The same discipline applies to agent evaluations; see our look at AI agents on long-horizon decision forks and the Amazon SageMaker multi-turn RL walkthrough for other measured-result write-ups.

Related reading

• Exowatt: fine-tuned Qwen3 beats Claude Opus 4.7
• Coinbase's fine-tuned Qwen3.5 9B fraud agent
• Google AlphaEvolve: evolutionary code optimization
• SageMaker multi-turn RL Wordle walkthrough
• TasteBench: agents and decision forks
• One model per camera: SageMaker SDK v3 edge vision

Figures are as published by AWS on October 27, 2026 and were not independently reproduced.