Documentation Scraper▌
by arabold
Easily retrieve swift language documentation from GitHub, NPM, PyPI, and web pages with accurate, up-to-date references
Provides specialized documentation scraping and retrieval from GitHub, NPM, PyPI, and web pages, enabling accurate reference to up-to-date library documentation without disrupting workflow.
Both formats append explainx.ai attribution and the canonical URL for this MCP server listing.
best for
- / Developers wanting AI assistants with current library knowledge
- / Teams needing accurate documentation references in AI workflows
- / Anyone tired of AI hallucinations about API details
capabilities
- / Scrape documentation from GitHub repositories
- / Index NPM and PyPI package documentation
- / Process HTML, Markdown, PDF, and Office documents
- / Query version-specific library documentation
- / Index local documentation folders
- / Search across multiple documentation sources
what it does
Fetches and indexes official documentation from GitHub, NPM, PyPI, and web sources so AI assistants can reference current, accurate library docs instead of hallucinating outdated information.
about
Documentation Scraper is a community-built MCP server published by arabold that provides AI assistants with tools and capabilities via the Model Context Protocol. Easily retrieve swift language documentation from GitHub, NPM, PyPI, and web pages with accurate, up-to-date references It is categorized under search web, developer tools.
how to install
You can install Documentation Scraper in your AI client of choice. Use the install panel on this page to get one-click setup for Cursor, Claude Desktop, VS Code, and other MCP-compatible clients. This server runs locally on your machine via the stdio transport.
license
MIT
Documentation Scraper is released under the MIT license. This is a permissive open-source license, meaning you can freely use, modify, and distribute the software.
readme
Grounded Docs: Your AI's Up-to-Date Documentation Expert
Docs MCP Server solves the problem of AI hallucinations and outdated knowledge by providing a personal, always-current documentation index for your AI coding assistant. It fetches official docs from websites, GitHub, npm, PyPI, and local files, allowing your AI to query the exact version you are using.

✨ Why Grounded Docs MCP Server?
The open-source alternative to Context7, Nia, and Ref.Tools.
- ✅ Up-to-Date Context: Fetches documentation directly from official sources on demand.
- 🎯 Version-Specific: Queries target the exact library versions in your project.
- 💡 Reduces Hallucinations: Grounds LLMs in real documentation.
- 🔒 Private & Local: Runs entirely on your machine; your code never leaves your network.
- 🧩 Broad Compatibility: Works with any MCP-compatible client (Claude, Cline, etc.).
- 📁 Multiple Sources: Index websites, GitHub repositories, local folders, and zip archives.
- 📄 Rich File Support: Processes HTML, Markdown, PDF, Word (.docx), Excel, PowerPoint, and source code.
🚀 Quick Start
1. Start the server (requires Node.js 22+):
npx @arabold/docs-mcp-server@latest
2. Open the Web UI at http://localhost:6280 to add documentation.
3. Connect your AI client by adding this to your MCP settings (e.g., claude_desktop_config.json):
{
"mcpServers": {
"docs-mcp-server": {
"type": "sse",
"url": "http://localhost:6280/sse"
}
}
}
See Connecting Clients for VS Code (Cline, Roo) and other setup options.
<details> <summary>Alternative: Run with Docker</summary>docker run --rm \
-v docs-mcp-data:/data \
-v docs-mcp-config:/config \
-p 6280:6280 \
ghcr.io/arabold/docs-mcp-server:latest \
--protocol http --host 0.0.0.0 --port 6280
</details>
🧠 Configure Embedding Model (Recommended)
Using an embedding model is optional but dramatically improves search quality by enabling semantic vector search.
Example: Enable OpenAI Embeddings
OPENAI_API_KEY="sk-proj-..." npx @arabold/docs-mcp-server@latest
See Embedding Models for configuring Ollama, Gemini, Azure, and others.
📚 Documentation
Getting Started
- Installation: Detailed setup guides for Docker, Node.js (npx), and Embedded mode.
- Connecting Clients: How to connect Claude, VS Code (Cline/Roo), and other MCP clients.
- Basic Usage: Using the Web UI, CLI, and scraping local files.
- Configuration: Full reference for config files and environment variables.
- Embedding Models: Configure OpenAI, Ollama, Gemini, and other providers.
Key Concepts & Architecture
- Deployment Modes: Standalone vs. Distributed (Docker Compose).
- Authentication: Securing your server with OAuth2/OIDC.
- Telemetry: Privacy-first usage data collection.
- Architecture: Deep dive into the system design.
🤝 Contributing
We welcome contributions! Please see CONTRIBUTING.md for development guidelines and setup instructions.
License
This project is licensed under the MIT License. See LICENSE for details.
FAQ
- What is the Documentation Scraper MCP server?
- Documentation Scraper is a Model Context Protocol (MCP) server profile on explainx.ai. MCP lets AI hosts (e.g. Claude Desktop, Cursor) call tools and resources through a standard interface; this page summarizes categories, install hints, and community ratings.
- How do MCP servers relate to agent skills?
- Skills are reusable instruction packages (often SKILL.md); MCP servers expose live capabilities. Teams frequently combine both—skills for workflows, MCP for APIs and data. See explainx.ai/skills and explainx.ai/mcp-servers for parallel directories.
- How are reviews shown for Documentation Scraper?
- This profile displays 72 aggregated ratings (sample rows for discoverability plus signed-in user reviews). Average score is about 4.6 out of 5—verify behavior in your own environment before production use.
Use Cases▌
Web Research & Information Gathering
Fetch and extract information from websites automatically
Example
Research competitor pricing, scrape product reviews, monitor news mentions
Automate 5-10 hours/week of manual web research
Content Monitoring & Alerts
Track website changes, new content, price updates
Example
Monitor competitor blog for new posts, track stock availability, watch for pricing changes
Stay informed without manual checking, never miss important updates
Data Extraction & Aggregation
Extract structured data from multiple websites
Example
Compile product listings from 10 e-commerce sites, aggregate job postings, collect real estate data
Build datasets 100x faster than manual copying
API-less Integration
Interact with services that don't offer APIs
Example
Check form submissions, validate website functionality, test user flows
Automate interactions with any website, even without API
Implementation Guide▌
Prerequisites
- ›Claude Desktop or Cursor with MCP support
- ›Understanding of web scraping ethics and robots.txt
- ›Rate limiting awareness to avoid overwhelming target sites
- ›Knowledge of legal restrictions on data collection
Time Estimate
20-40 minutes including configuration and testing
Installation Steps
- 1.Install web automation MCP server via npm or pip
- 2.Configure allowed domains and rate limits in MCP config
- 3.Test with simple fetch: 'Get content from example.com'
- 4.Progress to extraction: 'Extract all product prices from this page'
- 5.Set up monitoring: 'Check this URL daily for changes'
- 6.Parse structured data: 'Create CSV from this table'
- 7.Respect robots.txt and rate limits always
Troubleshooting
- ⚠403 Forbidden: Website blocks bots—respect their wishes, use official API instead
- ⚠Rate limit errors: Slow down requests, add delays between fetches
- ⚠Stale data: Target site changed HTML structure—update selectors
- ⚠Timeout errors: Site is slow or blocking—increase timeout, try different user agent
- ⚠JavaScript-rendered content: Use headless browser MCP servers for dynamic sites
Best Practices▌
✓ Do
- +Check robots.txt and respect crawl rules
- +Rate limit requests: 1-2 requests/second maximum
- +Use official APIs when available instead of scraping
- +Identify your bot with descriptive user agent
- +Cache results to minimize repeated requests
- +Handle errors gracefully with retries and fallbacks
- +Validate extracted data for accuracy
✗ Don't
- −Don't scrape sites that explicitly forbid it (robots.txt, ToS)
- −Don't overwhelm servers with rapid requests—use rate limiting
- −Don't scrape personal data without consent and legal basis
- −Don't ignore copyright on extracted content
- −Don't assume HTML structure is stable—handle changes
- −Don't use scraped data for commercial purposes without permission
💡 Pro Tips
- ★Use CSS selectors or XPath for robust data extraction
- ★Set up monitoring alerts for extraction failures (structure changed)
- ★Implement exponential backoff for retries on failures
- ★Store raw HTML for reprocessing if extraction logic changes
- ★Combine with data analysis tools for insights from extracted data
- ★Consider using official APIs or RSS feeds as more stable alternatives
Technical Details▌
Architecture
MCP server handles HTTP requests, HTML parsing, JavaScript rendering (if headless browser), and returns structured data to Claude.
Protocols
- HTTP/HTTPS
- WebSocket (for real-time sites)
- Puppeteer/Playwright (for JavaScript sites)
Compatibility
- Static HTML sites
- JavaScript-rendered SPAs (with headless browser)
- REST APIs
- GraphQL endpoints
When to Use This▌
✓ Use When
Use for research automation, content monitoring, data aggregation from multiple sources, and when official APIs don't exist. Best for read-only information gathering.
✗ Avoid When
Avoid for sites with APIs (use API instead), sites that explicitly forbid scraping, when data is copyrighted, or for login-required content without proper authorization.
Integration▌
- →Scheduled monitoring with change detection
- →Multi-source data aggregation pipelines
- →Fallback to web scraping when API rate limits hit
- →Headless browser for JavaScript-heavy sites
Discussion
Product Hunt–style comments (not star reviews)- No comments yet — start the thread.
List & Promote Your MCP Server
Share your MCP server with the developer community
Ratings
4.6★★★★★72 reviews- ★★★★★Jin Mensah· Dec 24, 2024
Useful MCP listing: Documentation Scraper is the kind of server we cite when onboarding engineers to host + tool permissions.
- ★★★★★Ava Chawla· Dec 24, 2024
Strong directory entry: Documentation Scraper surfaces stars and publisher context so we could sanity-check maintenance before adopting.
- ★★★★★Alexander Thomas· Dec 8, 2024
Documentation Scraper is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.
- ★★★★★Diego Zhang· Dec 8, 2024
We wired Documentation Scraper into a staging workspace; the listing’s GitHub and npm pointers saved time versus hunting across READMEs.
- ★★★★★James Verma· Dec 8, 2024
Documentation Scraper reduced integration guesswork — categories and install configs on the listing matched the upstream repo.
- ★★★★★Shikha Mishra· Dec 4, 2024
Documentation Scraper is a well-scoped MCP server in the explainx.ai directory — install snippets and categories matched our Claude Code setup.
- ★★★★★Amelia Nasser· Dec 4, 2024
Documentation Scraper is a well-scoped MCP server in the explainx.ai directory — install snippets and categories matched our Claude Code setup.
- ★★★★★Carlos Srinivasan· Nov 27, 2024
Documentation Scraper is a well-scoped MCP server in the explainx.ai directory — install snippets and categories matched our Claude Code setup.
- ★★★★★Carlos Singh· Nov 27, 2024
We evaluated Documentation Scraper against two servers with overlapping tools; this profile had the clearer scope statement.
- ★★★★★Yash Thakker· Nov 23, 2024
Documentation Scraper is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.
showing 1-10 of 72