A builder posting as @measure_plan trained a computer vision system to coach their own tennis game — tracking ball speed, forehand vs. backhand strokes, shot placement, and body position at the moment of contact, all from plain iPhone video. The September 9, 2026 post crossed 215K views, and the interesting part isn't the tennis use case specifically — it's how fast the underlying pipeline came together.
TL;DR
| Question | Answer |
|---|---|
| What does it track? | Ball speed, stroke type, shot placement, body position at contact |
| Input | Plain iPhone video — no special cameras or sensors |
| Detection model | A fine-tuned RF-DETR model |
| Labeling / fine-tuning | Roboflow Agent, using an AI model referred to as Astra 6 for data labeling |
| Pose estimation | MediaPipe (ViTPose named as an alternative) |
| Video annotation | Python + OpenCV |
| What made it fast | Claude Code integration with Roboflow Agent — auto-labeling and fine-tuning run entirely from the terminal |
| What's next | Actionable technique feedback, tracking progress across sessions over time |
The pipeline, piece by piece
The build breaks into four distinct stages, each handled by a different, purpose-built tool rather than one end-to-end model trying to do everything:
- Input — standard iPhone video of a practice session. No specialized hardware, which is the whole point: this is a pipeline anyone could point at their own footage.
- Detection — a fine-tuned RF-DETR model, Roboflow's real-time object detection, segmentation, and keypoint model, tuned specifically for tennis: the ball, racket, and court-relevant objects a general-purpose detector wouldn't be tuned for out of the box.
- Labeling and fine-tuning orchestration — Roboflow Agent, using an AI model referenced as Astra 6 for the data-labeling step, meaning the tedious part of any custom computer vision project — labeling enough training footage to fine-tune a detector — was itself AI-assisted rather than done by hand frame by frame.
- Body pose — MediaPipe for tracking body position at the moment of contact, with the builder noting ViTPose as a viable alternative in a reply to another builder asking the same question for a dance-sync project.
Video annotation — drawing the tracked overlays onto the output footage — runs through plain Python and OpenCV, the same combination that's underpinned custom computer vision tooling for years; the novelty here isn't in that layer, it's in how much of the harder upstream work (labeling, fine-tuning) got automated away.
Why "auto-labeling from the terminal" is the actual story
Custom object detection has historically had a brutal bottleneck that has nothing to do with model architecture: getting enough labeled training data. Fine-tuning a detector like RF-DETR for a specific sport requires a dataset of footage with the ball, players, and relevant objects manually annotated — historically a slow, tedious process done in a dedicated labeling UI, often the single most time-consuming part of any custom computer vision project, well beyond the actual model training step.
The builder's own account of what changed is specific: Roboflow Agent's Claude Code integration let them "auto-label / fine-tune RF-DETR entirely from the terminal," which they said made this project "a lot faster/smoother" than their previous sports computer vision projects. That's the pattern worth paying attention to, independent of tennis specifically — an agent (Claude Code) driving a labeling-and-training pipeline (Roboflow Agent) collapses what used to be several separate tools and manual handoffs (label in a UI, export, write a training script, iterate) into a single terminal-driven workflow.
Why this matters beyond one tennis project
This is a concrete instance of a broader shift explainx.ai has tracked across custom computer vision tooling in 2026: the gap between "I have an idea for a CV application" and "I have a working, fine-tuned model" has been shrinking fast, largely because the labeling and fine-tuning steps — previously the most manual, time-intensive part of the whole process — are increasingly agent-assisted rather than done by hand. Roboflow's own benchmarking work comparing VLMs reflects the same institutional focus: making it faster to go from raw footage to a working detector, not just building better detectors in isolation.
For anyone who has previously ruled out a custom computer vision project because fine-tuning a detector from scratch felt like a multi-week undertaking, this specific pipeline — Roboflow Agent for labeling/fine-tuning, Claude Code for orchestration, RF-DETR for detection, MediaPipe for pose — is a genuinely lower-friction starting stack than what was available even a year earlier in this same fast-moving space.
What people are already asking to add
The most common reaction in replies wasn't about the tracking accuracy — it was requests for actionable coaching feedback, not just tracking. The builder's own second post acknowledged this directly: filming the sessions surfaced footwork and backhand issues they hadn't consciously noticed, and the stated next step is adding technique feedback on top of the existing tracking, plus tracking progress across sessions over time rather than analyzing single clips in isolation.
That's a meaningfully harder problem than object detection and pose tracking — it requires the system to reason about what's wrong with a stroke and what to change, not just measure and report what happened. It's also exactly the kind of problem where a general-purpose reasoning model, fed structured tracking data (stroke type, ball speed, joint angles at contact) as context, is a natural fit layered on top of the existing computer vision pipeline rather than a separate system built from scratch.
What this means if you want to build something similar
- You don't need to build a custom object detector from zero anymore. Starting from a strong base model like RF-DETR and fine-tuning it for your specific domain, with agent-assisted labeling, is now a realistic weekend-scale project rather than a multi-week one.
- Pick your pose-estimation library based on what you're tracking, not just popularity. MediaPipe is a solid, well-documented default for general body pose; ViTPose is worth evaluating if your application needs more precision on specific joint positions, as the parallel dance-sync project in the replies was exploring.
- The labeling step is where agent tooling currently saves the most time. If you're evaluating whether an agent-assisted computer vision workflow is worth adopting for your own project, the labeling and fine-tuning orchestration step — not the final inference pipeline — is where you'll feel the biggest speedup.
- Plain iPhone video as input is a deliberate, replicable design choice, not a limitation — it's what makes a project like this something anyone could point at their own footage, rather than requiring specialized capture hardware most people don't have.
Why sports are a natural proving ground for this stack
Sports footage has a few properties that make it an unusually good testbed for this kind of agent-assisted computer vision pipeline, compared to more open-ended visual domains. Motion is fast and repetitive (the same stroke type happens dozens of times per practice session, giving a fine-tuning pipeline abundant similar examples to learn from), the relevant objects are few and well-defined (a ball, a racket, a court, a player's body), and there's an objective ground truth to validate against — a ball either crossed the line or it didn't, a stroke either connected cleanly or it didn't — which makes it much easier to spot-check whether a fine-tuned detector is actually working than it would be for a more subjective or ambiguous visual task.
That combination — abundant repetitive examples, a small well-defined object set, and checkable ground truth — is exactly the profile that makes a fine-tuning-from-agent-labeled-data workflow shine. It's a large part of why sports and movement tracking (this tennis project, the parallel dance-sync project mentioned in the replies, and prior computer vision projects tackling basketball shot tracking or golf swing analysis) keep showing up as some of the most common and successful hobbyist computer vision builds: the domain constraints do a lot of the work that would otherwise require much larger, more carefully curated datasets in a messier visual domain.
The gap between tracking and coaching
It's worth being precise about where this project currently sits versus where it's headed, because the distinction matters for anyone evaluating whether to build something similar. What's built today is a measurement system: it can tell you your ball speed, classify your stroke type, and show you your body position at contact. What it can't yet do is coach — tell you specifically that your backhand contact point is too far behind your body, or that your footwork isn't getting you into position early enough for a particular shot.
That gap between measurement and coaching is where most of the remaining hard engineering work lives, and it's a genuinely different kind of problem than the detection and pose-estimation work already done. It likely requires either a domain-specific rules engine (encoding known biomechanical principles about what "good" technique looks like for each stroke type) or a general-purpose reasoning model fed the structured tracking output as context and asked to compare it against known good technique — plausibly some combination of both, since pure LLM reasoning over raw joint-angle numbers without domain-specific grounding tends to produce plausible-sounding but not necessarily accurate technical feedback in physical-skill domains like this one.
What to watch next
- Whether the builder ships the planned app with progress tracking and actionable coaching feedback, and what that feedback layer actually looks like technically.
- Whether similar Roboflow Agent + Claude Code pipelines get applied to other sports or movement-tracking domains, following the same pattern the dance-sync reply already suggests is happening in parallel.
- Broader adoption of agent-driven labeling/fine-tuning workflows for custom computer vision projects outside of sports specifically.
Related reading
- RF-DETR: Real-Time Object Detection, Segmentation, and Keypoints
- Roboflow Benchmarks GPT-5.6 Sol and Gemini 3.5 Flash as VLMs
- Frigate NVR: An AI Camera System Complete Guide
- Coral Edge AI Platform: The Complete Guide
- A 4B Open-Source VLM Reportedly Beats Qwen 122B on GeoGuessr-Style Benchmarks
This post reflects @measure_plan's public posts on X as of September 9-10, 2026. Model names, tools, and workflow details are as described by the builder; the app itself has not shipped publicly at the time of writing.