Coding agents can read code, screenshots and logs, but sound has been a blind spot. earshot, a repo by Leonxlnx that was trending on GitHub with about 1,500 stars when we checked, is an attempt to fix that. Per its README on GitHub, it is a CLI, Python API and MCP server that runs on CPU and turns sound into "evidence an agent can reason about."
It is MIT-licensed, uses two open audio models, and works offline after a first-run download. Below: what it does, how to set it up, the author's accuracy claims, and the limits the README states itself.
Quick reference
| Question | Answer |
|---|---|
| What is it? | CLI, Python API and MCP server that lets agents "hear" audio |
| Needs a GPU? | No, CPU only |
| Needs an API key? | Not for the core listening models; the install includes an optional llm extra |
| Install size | About 1.1 GB of models on the first run, then offline |
| License | Code MIT; models keep their own licenses (Apache-2.0 and BSD-3-Clause) |
| Stars | About 1.5k (90 forks, 1 open issue, 2 open pull requests at fetch time) |
| Best for | Sound design, audio QA, scoring video soundtracks, agent loops that make sounds |
| Biggest caveat | Not a judge of taste, emotion or speech; can overfit its own models |
What does earshot actually do?
According to the README, earshot covers five jobs:
- Describe audio in words using tags from an AudioSet classifier.
- Score fit to a text description, so an agent can ask whether a clip sounds like "a wooden door slamming."
- Find the best moments in a long recording.
- Flag real versus code-synthesized sound.
- Analyze film audio: it produces listening transcripts and annotated spectrogram images, and scores launch videos against seven reference films.
On the making side, earshot forge writes sounds as code, using physical synthesis recipes, then scores and refines them with the same listening models. earshot fit turns a recording into a synthesis recipe that re-renders it. There is a 201-sound bank that renders effects by ID with no models or network, and earshot kit builds per-film sound kits.
The idea is a closed loop: an agent writes a sound as code, listens to it, compares it with the target description, and revises. That is the same pattern as an agent running tests, applied to audio.
Audio waveform lines, representing how earshot lets coding agents hear audio
What would an agent loop with ears look like?
The README describes the pieces; here is how they fit together in practice, as explainx.ai's read rather than a documented workflow. An agent receives a request such as "make a short, soft UI click". It calls the forge to write a synthesis recipe in code, renders the sound, and asks the listening models two questions: how well does this match the description, and does it look like a real recording or a synthesized one? If the score is low, the agent edits the recipe and tries again. When the score is high, it hands the result to a person.
Two design choices make this workable. First, everything is numeric, so the agent gets a score it can compare across attempts instead of a vague impression. Second, it runs on CPU and offline after setup, so a loop of dozens of attempts costs time but not API fees. The cost is latency: model reloads of 20 to 40 seconds per CLI call are the reason the README steers repeated use toward the MCP server, which keeps the models loaded.
A raindrop striking a bowl and making ripple rings, representing sound effect generation and scoring with earshot
What are people asking about earshot?
Can it transcribe speech? No. The README lists speech content among the things it cannot judge. It can tell you a clip contains speech-like sound, not what was said.
Does it send my audio anywhere? The core listening models run locally on CPU and work offline after the first download, per the README. The install includes an llm extra, so check what that option contacts before using it on private recordings.
Is a 98% claim safe to rely on? It is measured on a held-out library chosen by the author. Test it on your own sounds, especially quiet or layered ones, where the README itself says tagging is weaker.
Why use a text-audio model for judging at all? Because a similarity score between a clip and a sentence is something an agent can compute and compare. It is a proxy for "sounds like", not a measure of "sounds good".
Which models power it?
Two open models do the listening, per the README:
- CLAP (
laion/larger_clap_general, Apache-2.0) provides text-audio embeddings. The model card on Hugging Face describes CLAP as "to audio what CLIP is to image": it is trained on paired audio and text, with a Swin Transformer audio branch and a RoBERTa text branch projecting into a shared space, so a dot product gives a similarity score. It supports zero-shot classification against text labels you supply. - AST (
MIT/ast-finetuned-audioset-10-10-0.4593, BSD-3-Clause) provides AudioSet tagging.
Neither is new. The novelty is wiring them into agent-friendly commands and an MCP server. That means earshot inherits their strengths and weaknesses, including the limits of a model trained on general audio.
How do I install and use it?
Requirements: Python 3.10+, ffmpeg and ffprobe on your PATH, CPU only. The README's flow is a git clone, a virtualenv, a CPU build of torch, then pip install -e ".[mcp,llm,dev]", and finally a health check:
git clone https://github.com/Leonxlnx/earshot
cd earshot
python -m venv .venv
pip install -e ".[mcp,llm,dev]"
earshot doctor --load
Install the CPU build of torch first, as the README directs, before the editable install.
The first run downloads about 1.1 GB of models into a local cache; afterward it works offline. Each CLI call reloads the models, which the README puts at about 20 to 40 seconds, so for repeated use it recommends the MCP server instead. Connecting it to Claude Code is a single claude mcp add command (see the README for the exact line), and two agent skills ship in the skills/ folder.
If you are new to the pieces, our guides to MCP and agent skills cover how a server and a skill plug into an agent, and the Claude Code commands reference covers the surrounding commands.
How accurate are its claims?
All of these are the author's numbers from the README, not independent tests:
| Claim | Number | Caveat from the README |
|---|---|---|
| "Strong" match verdict is the right kind of sound | 98% on a held-out library | Held-out by the author |
| Real versus synthesized detection | ROC-AUC 0.92 held-out | Measured on the author's data |
| Blind test: author picks the made sound | 8 of 15 correct | The README calls it "a coin flip" on a small test |
The blind test result is the most useful one. It says that, at least in that small trial, synthesized sounds from the forge were hard for a human to tell from real ones, which is the success case for a sound-design tool. It is also a reminder that fifteen trials is a very small sample.
What are its limits?
The README is candid. It says earshot is reliable for loudness, timing, sync to picture, and judging single sounds. It is less reliable for tags inside busy mixes, genre and mood. It cannot judge taste, emotion, provenance, licensing, speech content, or content above 14 kHz.
Two further cautions follow from how it works:
- Overfitting to its own ears. The README warns that the forge's search can overfit earshot's own models: if the same models both generate and judge, a sound can score well without sounding good. It recommends human listening tests.
- The launch scorecard has a style bias. Scoring a video against seven reference films reflects those seven films, not all good films.
Where does it fit among agent tools?
Agent skills that make media are a growing category on this blog. Lemo-Opuscar makes short films with code instead of a video model; earshot adds the missing sense, so an agent making a film can also check its soundtrack. For video-to-sound tools, see Mirelo SFX 2.0, and for on-device speech recognition see Cactus Whistle. Note that earshot does not do speech content: for transcripts you still need a speech-to-text tool.
Should you try it?
Try it if you build agent loops that generate or check audio: game and UI sound effects, soundtrack QA, or scoring many clips against descriptions. Skip it if you need judgments about music quality, emotion, speech, or licensing, which it says it cannot give. Treat the accuracy figures as a starting point to verify on your own audio, and keep a human listening pass in the loop.
A sensible first test: pick ten clips you already know well, run them through the describe and match commands, and see whether the verdicts agree with your own ears. If they do on clean single sounds and drift on busy mixes, that matches the README's own warning, and you will know where to trust it.
Details reflect the earshot README and the CLAP model card as of October 20, 2026. Star counts and install steps change; check the repo.