explainx / blog
LLMs do not natively watch most video files. Gemini samples frames; Claude needs prep. Solutions: scene-aware frame packs (claude-real-video), transcripts (video-use), native APIs, and DIY ffmpeg — compared with costs.
Jun 27, 2026
Generative AI is the technology behind ChatGPT, Midjourney, and GitHub Copilot — AI that creates new content rather than just analyzing existing data. This foundational guide explains what it is, how it works in plain English, and how to start using it practically, even if you have zero technical background.
Jun 15, 2026
The open-source model landscape in 2026 has closed the frontier gap to single digits on most benchmarks. This guide matches each major closed-source model—GPT-5.5, Claude Opus 4.8, Claude Fable 5, Gemini 3.1 Pro, o3, GPT-4o—with its strongest open-weight local alternative, with real benchmark numbers, true cost comparisons, and honest notes on where proprietary models still hold an edge.
Jun 23, 2026
The standard assumption for running a 70B model locally: you need 140GB of VRAM. AirLLM breaks that assumption by loading layers one at a time from disk, holding only one layer in GPU memory at any moment. 21K+ GitHub stars, three lines of code to start. Here is what it actually buys you and what it costs you.
Can Claude watch a video? Not the way humans do. Paste a YouTube URL into ChatGPT and you mostly get captions, not frame-by-frame vision. Upload an MP4 to Claude and you often hit unsupported file type. Even Gemini, which accepts video natively, typically samples frames at a fixed rate — fine for a lecture, weak for a 15-second reel with six cuts.
The workable pattern in 2026 is not "give the model the video." It is give the model what matters from the video — speech as text, visuals as deduplicated keyframes, metadata as a manifest — then ask your questions.
| Approach | Best for | Visual fidelity | Runs locally? | Typical cost |
|---|---|---|---|---|
| Gemini native video (API / AI Studio) | Quick analysis, Google stack | ~1 fps default; misses fast cuts | No — cloud upload | Gemini token pricing |
| ChatGPT + YouTube link | Spoken content, summaries | Transcript-first; weak on B-roll | No | Plus / API |
claude-real-video (crv) | URLs + local files; scene-heavy edits | Scene changes + dedup | Yes | Free tool + LLM tokens |
| video-use + Claude Code | Editing (cuts, grade, subs) | Transcript-first; composites on demand | Partial (local ffmpeg) | Scribe API + Claude |
| DIY ffmpeg + Whisper | Full control, air-gapped | Whatever you script | Yes | Your time |
| Video VLMs (Marlin) | Structured events from feeds | Model-specific | Self-host option | GPU + ops |
Bottom line: For Q&A and research, use frames + transcript + manifest. For editing, use video-use. For fastest cloud try, use Gemini video. For privacy and reels, use local prep (crv or DIY).
Multimodal models do not stream 24 fps into context. They receive a bounded set of images and text inside a token budget.
| Failure mode | What happens |
|---|---|
| Fixed-interval sampling (1 frame/sec) | 10-minute static screencast → ~600 near-duplicate frames; 30-second reel → misses cuts between samples |
| Transcript-only (YouTube in ChatGPT) | Answers dialogue; blind to on-screen code, charts, or lip-sync fraud |
| Raw frame dump | 1 minute × 30 fps × ~1,500 tokens/image → millions of tokens — slow, expensive, noisy |
| No video ingest (Claude file upload) | You must preprocess before the model sees anything |
video-use's core insight applies broadly: the LLM should read structure, not raw pixels — but for open-ended video Q&A you need both transcript and selectively chosen frames, not transcript alone.
Google's Gemini models accept video in AI Studio and the Gemini API. Default sampling is often ~1 frame per second — documented behavior teams hit when analyzing long files.
Good for: meeting recordings, lectures, slow-paced demos.
Weak for: TikTok edits, sports highlights, music videos with fast cuts.
See Gemini Omni Flash for Google's video stack direction — generation and understanding are related but not identical products.
Useful when spoken content carries the signal. Weak when the answer is on screen but never spoken — UI walkthroughs, silent demos, visual gags.
Cost: bundled in subscription or API vision pricing; you do not control frame selection.
claude-real-video)claude-real-video (crv, MIT, ~250 GitHub stars as of July 2026) is one open-source answer when you want any LLM — Claude, ChatGPT, Gemini, local models — to reason over what changed visually, without uploading the video to a vendor.
pip install claude-real-video
pip install "claude-real-video[whisper]" # + audio transcription
brew install ffmpeg # macOS; required on all platforms
crv "https://www.youtube.com/watch?v=..."
# → crv-out/frames/*.jpg
# → crv-out/transcript.txt
# → crv-out/MANIFEST.txt
| Fixed 1 fps | crv | |
|---|---|---|
| Static 10-minute slide | ~600 similar frames | Collapses to few frames after dedup |
| Fast-cut reel | Misses between samples | Catches scene changes |
| A-B-A edit (repeat shot) | Sends A twice | Sliding-window dedup sends each shot once |
| Audio | Often ignored | Whisper or embedded subtitles |
| Privacy | Often cloud upload | Stays on your machine |
How it works (short): yt-dlp or local file → ffmpeg scene-change + fps floor → pixel-diff dedup (not perceptual hash — hashes miss flat-color hue shifts) → subtitle track if present else Whisper → MANIFEST.txt for the model.
Tuning flags that matter:
| Flag | Default | Effect |
|---|---|---|
--scene | 0.30 | Lower = more frames |
--max-frames | 150 | Hard cap |
--dedup-threshold | 8 | Higher = fewer frames kept |
--dedup-window | 4 | Stops A-B-A repeat sends |
--report | off | HTML report of keep/drop decisions |
Optional --keep-audio: saves audio.m4a for models that accept audio (Gemini, GPT-4o audio) when tone and music matter — transcript alone loses both.
crvYou are analyzing a video I preprocessed locally.
Read MANIFEST.txt for metadata, transcript.txt for speech, and frames/*.jpg in order.
Questions:
1. What is the main argument in the first 2 minutes?
2. Which frame shows the pricing table?
3. List every scene change topic in chronological order.
Drop the folder into Claude Projects, paste paths in ChatGPT, or attach images in Gemini — same artifacts, any frontier model.
video-use)If your goal is produce a new video, not understand an existing one, use video-use:
takes_packed.mdtimeline_view composites for cut decisionsThe LLM never watches — it reads and edits. Perfect for launch cuts and filler removal (Fable 5 launch pipeline); wrong tool for "summarize this documentary's visual motifs."
Pairing: Run crv for research Q&A; run video-use when you need final.mp4.
Minimal version without crv:
# Scene frames
ffmpeg -i input.mp4 -vf "select='gt(scene,0.3)',showinfo" -vsync vfr frames/%04d.jpg
# Audio
whisper input.mp4 --model medium --output_format txt
You lose sliding-window dedup, manifest generation, and URL fetch — but keep air-gapped control. Reasonable for one-off internal compliance reviews.
For camera feeds and agent loops ("what changed since last frame?"), small video VLMs like Marlin 2B on NemoStation target structured outputs — not consumer YouTube Q&A.
Use when building products, not when a marketer wants a one-off summary of a webinar.
Naive approach: 30 fps × 60 sec × 1,500 tokens ≈ 2.7M tokens/minute of vision — unusable.
crv on a 8-minute talk (illustrative):
| Stage | Output |
|---|---|
| Scene + floor extraction | ~90 raw frames |
| Dedup at threshold 8 | ~25–40 kept frames |
| Transcript | ~2,000 words (~2,500 tokens) |
| Vision tokens (varies by resize) | ~40k–100k total |
You trade minutes of local ffmpeg for orders-of-magnitude fewer vision tokens — the right trade for token budget planning.
| Cost line | Local crv | Gemini native upload |
|---|---|---|
| Tooling | $0 (MIT) | API usage |
| Compute | Your CPU/GPU for Whisper | Google-side |
| Privacy | No video leaves disk | Video uploaded |
| LLM bill | Lower context | Higher if re-sending frames |
| Your job | Pick |
|---|---|
| "Summarize this YouTube lecture" | Gemini native or crv + any LLM |
| "Find when the UI bug appears" | crv (scene frames) |
| "Cut ums and ship launch video" | video-use |
| "Compliance — air-gapped" | DIY or crv offline |
| "Build a video-watching agent product" | Marlin / custom VLM |
crv is young — v0.1.x, no formal releases yet; verify on your content types before production pipelines.--cookies is for your authenticated access, not credential sharing.Tool versions accurate as of July 3, 2026. claude-real-video: github.com/HUANGCHIHHUNGLeo/claude-real-video. Always verify current Gemini sampling defaults in Google AI documentation.