A developer's account of what happened after a Hacker News launch is a useful warning for anyone running a small site. In a post titled "I posted to HN, and then my website was flooded with AI BOT traffic" on their personal blog, the author of the feed reader sprinklz.io describes how a post that earned about a hundred upvotes was followed by days of slowness. Logs showed that most requests never rendered the page, and nearly half of all traffic came from a single service that describes itself as listening for website changes and summarizing them into an AI newsletter. It is one site's anecdote, and a Hacker News thread pushed back on some of it, but the pattern and the defenses are worth knowing. This post summarizes the account, what official crawler documentation says, and a checklist.
It builds on our reporting on AI and bot traffic overtaking humans and Cloudflare's breakdown of search, agent and training traffic.
Quick reference
| Question | Answer |
|---|---|
| What happened? | After an HN post (about 100 upvotes), the site became very slow for days |
| What did the logs show? | Most requests never fetched articles or ran the SPA JavaScript |
| Biggest offender? | Nearly half of traffic from one change-monitoring AI newsletter service (unnamed) |
| Other traffic | Self-identified crawlers (Claude, OpenAI, Google), plus vulnerability scanners probing paths like a WordPress file on a non-WordPress site |
| Fixes | Blocked the abuser, robots.txt for AI scanners, rate limits, performance work |
| Caveat | Single-site account; one HN commenter saw tiny load from an HN number-one post |
What did the developer actually find?
The author's account is short and specific. Slowness did not fade after the HN spike; it got worse several days later, and the metrics did not match: a handful of real logged-in users but a server under heavy load. They added top-level logging per endpoint and found that most requests never tried to fetch news articles. Since the site is a single-page app, that meant most requests never ran the JavaScript that renders the page, which is the signature of scripted fetchers rather than browsers.
Aggregate NGINX logs showed that nearly half of traffic came from one user, which the author looked up and found described itself as a service that watches sites for changes and summarizes them into an AI newsletter, querying "many many times per-minute." They blocked it. Other traffic came from crawlers that identified themselves, including Claude's and OpenAI's bots and Google's indexer, and from vulnerability scanners requesting paths such as a WordPress file on a site that is not WordPress.
The remedies: a robots.txt that discourages most AI scanners but allows Google and search traffic, probing their own site with the scanner paths (using an AI assistant) to ensure nothing leaked, reasonable rate limits, and several weeks of infrastructure improvements so the site can handle much more real traffic.
What did the HN thread add?
The thread had modest engagement (35 points when we read it) and a mix of reactions.
- One commenter, vanschelven, reported that their own HN post reached number one and had only a single-digit percentage effect on CPU and network on a roughly five-euro-a-month VPS, arguing that the old C10k problem was solved long ago. That is a useful counterpoint: HN traffic itself is rarely the culprit, and heavy per-request work or bot loops are.
- Another, bix6, asked which service the author meant; the post deliberately does not name it, so we cannot either.
- oneeyedpigeon said they were confused that the blog is a single page containing every post. That is the commenter's observation; our own aside is that page weight and structure affect how costly each fetch is.
We present these as commenters' reports, not verified facts.
How are the main AI crawlers documented?
Not all AI traffic is the same. Two vendors publish clear documentation.
OpenAI documents three agents: GPTBot, which crawls content that may be used for training (disallowing it signals content should not be used for training); OAI-SearchBot, which surfaces sites in ChatGPT search (opted-out sites will not be shown in search answers, though they can appear as links); and ChatGPT-User, which visits pages when a user asks ChatGPT or a custom GPT a question. OpenAI states that ChatGPT-User is not used for automatic crawling and that, because users trigger the visits, "robots.txt rules may not apply." Each bot is controlled independently.
Anthropic documents three bots: ClaudeBot (content that may contribute to training), Claude-User (retrieval when a user asks a question) and Claude-SearchBot (improves search results). Blocking is by robots.txt per subdomain, and Anthropic warns that blocking its IP addresses may not work reliably because the bots then cannot read your robots.txt. It also supports the non-standard Crawl-delay extension.
The key point: these are the polite, identified actors. The service the developer blocked and the vulnerability scanners are not covered by any vendor's opt-out.
A defense checklist for a small site
- Log by user agent and path before changing anything. The developer's breakthrough was counting requests per endpoint and finding most never fetched real content. A simple aggregate of the NGINX log by user agent and IP shows who is heavy.
- Decide per bot what you want. Allow search agents if you want visibility in AI answers, disallow training agents if you want to opt out. Write a separate robots.txt rule per user agent.
- Add Crawl-delay for bots that support it.
- Rate-limit at the edge. Per-IP and per-user-agent limits at the proxy or CDN stop clients that ignore robots.txt. Return 429 with a Retry-After header.
- Block clearly abusive clients. If one client makes many requests per minute for the same content, block its address range or user agent.
- Make pages cheap. Cache static responses, serve pre-rendered HTML for content pages, and avoid dynamic work for unauthenticated requests.
- Test your own exposure. The author used an assistant to query scanner paths and mimic scanners to confirm nothing leaked. Do the same for admin paths and debug files.
- Do not rely on IP-blocking official crawlers. Per Anthropic, that can stop them from reading robots.txt and defeat opt-outs.
A minimal robots.txt that opts out of training but allows search and user-triggered visits looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
Enforcement beats etiquette: a rate-limit example
robots.txt is a request, not a wall. For clients that do not honor it, enforcement belongs at the proxy. A minimal NGINX pattern limits each client address to a modest request rate and returns 429 when it is exceeded:
limit_req_zone $binary_remote_addr zone=perip:10m rate=5r/s;
server {
location / {
limit_req zone=perip burst=20 nodelay;
limit_req_status 429;
}
}
Tune the numbers to your real traffic: look at what a busy human session looks like in your logs and set the limit comfortably above it. For a single-page app, apply a stricter limit to the API paths than to static assets, because API calls are what cost you CPU. If a client keeps hitting the limit, block it by address range or user agent, and consider a CDN with bot-management features if you do not want to maintain rules yourself.
Remember that rate limits shift cost onto the bots, not away from you: a client that is refused still opens a connection. Serving cached responses for the most requested pages is the cheapest long-term defense, because it makes each fetch nearly free whether the visitor is a person or a scraper.
What this means for builders
If you plan a launch, assume bots will arrive with the humans. Our coverage of Google-zero fears and publishers threatening to block crawlers shows the commercial side: publishers weigh visibility against being scraped. For tiny sites, the cost is performance and hosting bills, and the answer is mostly engineering: measure, rate-limit, cache. The same pressure shows up in other places where AI output overwhelms human-scale systems, such as arXiv capping authors at two submissions a month.
If you build agents that fetch pages, be a good citizen: identify yourself with a clear user agent, honor robots.txt where you can, cache results, and rate-limit your own requests. If you run agents with web access, AgentBeam is the agent security platform from the explainx.ai team that stops AI agents before they take dangerous actions.
Caveats
- This is one developer's account, without published logs.
- The offending service is unnamed, so its claims cannot be checked.
- Impact varies widely: another commenter saw minimal load from the same kind of HN spike.
- Vendor documentation describes intended behavior, not guaranteed behavior.
What we are watching next
- Whether CDNs and hosts add default AI-bot controls for small sites.
- Whether change-monitoring newsletter services publish crawl policies or identify themselves.
- Updates to OpenAI and Anthropic crawler documentation, including any new user agents.
Related reading
- Expand Fetch: an agent web-reading API with a self-run 74% benchmark claim
- AI and bot traffic overtook humans
- Cloudflare AI traffic: search, agent, training
- Google-zero publishers and crawler blocking
- arXiv's two-submission cap
- Cloudflare web search API for agents
- Cloudflare Sandbox SDK 1.0 for agents
Details are accurate as of October 13, 2026. The traffic figures come from the author's own post; crawler behavior is as documented by OpenAI and Anthropic at the time of writing.