by thedaviddias
Explore websites using llms.txt files with LLMs.txt Explorer. Fetch and parse site-specific language model instructions
Discovers and fetches llms.txt files from websites, which contain instructions for how AI models should interact with those sites.
LLMs.txt Explorer is a community-built MCP server published by thedaviddias that provides AI assistants with tools and capabilities via the Model Context Protocol. Explore websites using llms.txt files with LLMs.txt Explorer. Fetch and parse site-specific language model instructions It is categorized under search web.
You can install LLMs.txt Explorer in your AI client of choice. Use the install panel on this page to get one-click setup for Cursor, Claude Desktop, VS Code, and other MCP-compatible clients. This server runs locally on your machine via the stdio transport.
NOASSERTION
LLMs.txt Explorer is released under the NOASSERTION license.
Fetch and extract information from websites automatically
Example
Research competitor pricing, scrape product reviews, monitor news mentions
Automate 5-10 hours/week of manual web research
Track website changes, new content, price updates
Example
Monitor competitor blog for new posts, track stock availability, watch for pricing changes
Stay informed without manual checking, never miss important updates
Extract structured data from multiple websites
Example
Compile product listings from 10 e-commerce sites, aggregate job postings, collect real estate data
Build datasets 100x faster than manual copying
Share your MCP server with the developer community
LLMs.txt Explorer is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.
LLMs.txt Explorer is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.
Strong directory entry: LLMs.txt Explorer surfaces stars and publisher context so we could sanity-check maintenance before adopting.
LLMs.txt Explorer is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.
Strong directory entry: LLMs.txt Explorer surfaces stars and publisher context so we could sanity-check maintenance before adopting.
Strong directory entry: LLMs.txt Explorer surfaces stars and publisher context so we could sanity-check maintenance before adopting.
Useful MCP listing: LLMs.txt Explorer is the kind of server we cite when onboarding engineers to host + tool permissions.
LLMs.txt Explorer reduced integration guesswork — categories and install configs on the listing matched the upstream repo.
Useful MCP listing: LLMs.txt Explorer is the kind of server we cite when onboarding engineers to host + tool permissions.
I recommend LLMs.txt Explorer for teams standardizing on MCP; the explainx.ai page compares cleanly with sibling servers.
showing 1-10 of 50
A Model Context Protocol server for exploring websites with llms.txt files. This server helps you discover and analyze websites that implement the llms.txt standard.
check_website - Check if a website has llms.txt files
list_websites - List known websites with llms.txt files
Install dependencies:
pnpm install
Build the server:
pnpm run build
For development with auto-rebuild:
pnpm run watch
To install mcp-llms-txt-explorer for Claude Desktop automatically via Smithery:
npx -y @smithery/cli install @thedaviddias/mcp-llms-txt-explorer --client claude
To use this server:
# Clone the repository
git clone https://github.com/thedaviddias/mcp-llms-txt-explorer.git
cd mcp-llms-txt-explorer
# Install dependencies
pnpm install
# Build the server
pnpm run build
To use with Claude Desktop, add the server config:
On MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json
On Windows: %APPDATA%/Claude/claude_desktop_config.json
{
"mcpServers": {
"llms-txt-explorer": {
"command": "node",
"args": ["/path/to/llms-txt-explorer/build/index.js"]
}
}
}
For npx usage, you can use:
{
"mcpServers": {
"llms-txt-explorer": {
"command": "npx",
"args": ["-y", "@thedaviddias/mcp-llms-txt-explorer"]
}
}
}
Since MCP servers communicate over stdio, debugging can be challenging. We recommend using the MCP Inspector, which is available as a package script:
pnpm run inspector
The Inspector will provide a URL to access debugging tools in your browser.
This project is licensed under the MIT License—see the LICENSE file for details.
Interact with services that don't offer APIs
Example
Check form submissions, validate website functionality, test user flows
Automate interactions with any website, even without API
Prerequisites
Time Estimate
20-40 minutes including configuration and testing
Steps
Troubleshooting
✓ Do
✗ Don't
💡 Pro Tips
Architecture
MCP server handles HTTP requests, HTML parsing, JavaScript rendering (if headless browser), and returns structured data to Claude.
Protocols
Compatibility
✓ Use when
Use for research automation, content monitoring, data aggregation from multiple sources, and when official APIs don't exist. Best for read-only information gathering.
✗ Avoid when
Avoid for sites with APIs (use API instead), sites that explicitly forbid scraping, when data is copyrighted, or for login-required content without proper authorization.