Web Fetch▌
by kwp-lab
Web Fetch is a web scraping tool that converts web pages to markdown, extracts images, and works with proxies for secure
Fetches and converts web pages to markdown format with automatic image extraction and proxy support for accessing content through corporate networks or restricted environments.
Both formats append explainx.ai attribution and the canonical URL for this MCP server listing.
best for
- / Corporate environments with proxy requirements
- / Content research and web scraping
- / Converting web articles for analysis
capabilities
- / Fetch web pages and convert to markdown
- / Extract image URLs from web content
- / Route requests through HTTP/HTTPS proxies
- / Access content through corporate firewalls
what it does
Fetches web pages and converts them to markdown format while extracting image URLs. Includes proxy support for corporate networks and restricted environments.
about
Web Fetch is a community-built MCP server published by kwp-lab that provides AI assistants with tools and capabilities via the Model Context Protocol. Web Fetch is a web scraping tool that converts web pages to markdown, extracts images, and works with proxies for secure It is categorized under search web.
how to install
You can install Web Fetch in your AI client of choice. Use the install panel on this page to get one-click setup for Cursor, Claude Desktop, VS Code, and other MCP-compatible clients. This server runs locally on your machine via the stdio transport.
license
MIT
Web Fetch is released under the MIT license. This is a permissive open-source license, meaning you can freely use, modify, and distribute the software.
readme
MCP Fetch
Model Context Protocol server for fetching web content with custom http proxy. This allows Claude Desktop (or any MCP client) to fetch web content and handle images appropriately.
<a href="https://glama.ai/mcp/servers/@kwp-lab/mcp-fetch"> <img width="380" height="200" src="https://glama.ai/mcp/servers/@kwp-lab/mcp-fetch/badge" /> </a>This repository forks from the @smithery/mcp-fetch and replaces the node-fetch implementation with the library node-fetch-native.
The server will use the http_proxy and https_proxy environment variables to route requests through the proxy server by default if they are set.
You also can set the MCP_HTTP_PROXY environment variable to use a different proxy server.
Available Tools
fetch: Retrieves URLs from the Internet and extracts their content as markdown. If images are found, their URLs will be included in the response.
Image Processing Specifications:
Only extract image urls from the article content, and append them to the tool result:
{
"params": {
"url": "https://www.example.com/articles/123"
},
"response": {
"content": [
{
"type": "text",
"text": "Contents of https://www.example.com/articles/123:
Here is the article content
Images found in article:
- https://www.example.com/1.jpg.webp
- https://www.example.com/2.jpg.webp
- https://www.example.com/3.webp"
}
]
}
}
Quick Start (For Users)
To use this tool with Claude Desktop, simply add the following to your Claude Desktop configuration (~/Library/Application Support/Claude/claude_desktop_config.json):
{
"tools": {
"fetch": {
"command": "npx",
"args": ["-y", "@kwp-lab/mcp-fetch"],
"env": {
"MCP_HTTP_PROXY": "https://example.com:10890" // Optional, remove if not needed
}
}
}
}
This will automatically download and run the latest version of the tool when needed.
Required Setup
- Enable Accessibility for Claude:
- Open System Settings
- Go to Privacy & Security > Accessibility
- Click the "+" button
- Add Claude from your Applications folder
- Turn ON the toggle for Claude
For Developers
The following sections are for those who want to develop or modify the tool.
Prerequisites
- Node.js 18+
- Claude Desktop (install from https://claude.ai/desktop)
- tsx (install via
npm install -g tsx)
Installation
Installing via Smithery
To install MCP Fetch for Claude Desktop automatically via Smithery:
npx -y @smithery/cli install @kwp-lab/mcp-fetch --client claude
Manual Installation
git clone https://github.com/kwp-lab/mcp-fetch.git
cd mcp-fetch
npm install
npm run build
Configuration
-
Make sure Claude Desktop is installed and running.
-
Install tsx globally if you haven't:
npm install -g tsx # or pnpm add -g tsx -
Modify your Claude Desktop config located at:
~/Library/Application Support/Claude/claude_desktop_config.json
You can easily find this through the Claude Desktop menu:
- Open Claude Desktop
- Click Claude on the Mac menu bar
- Click "Settings"
- Click "Developer"
Add the following to your MCP client's configuration:
{
"tools": {
"fetch": {
"args": ["tsx", "/path/to/mcp-fetch/index.ts"]
}
}
}
FAQ
- What is the Web Fetch MCP server?
- Web Fetch is a Model Context Protocol (MCP) server profile on explainx.ai. MCP lets AI hosts (e.g. Claude Desktop, Cursor) call tools and resources through a standard interface; this page summarizes categories, install hints, and community ratings.
- How do MCP servers relate to agent skills?
- Skills are reusable instruction packages (often SKILL.md); MCP servers expose live capabilities. Teams frequently combine both—skills for workflows, MCP for APIs and data. See explainx.ai/skills and explainx.ai/mcp-servers for parallel directories.
- How are reviews shown for Web Fetch?
- This profile displays 54 aggregated ratings (sample rows for discoverability plus signed-in user reviews). Average score is about 4.6 out of 5—verify behavior in your own environment before production use.
Use Cases▌
Web Research & Information Gathering
Fetch and extract information from websites automatically
Example
Research competitor pricing, scrape product reviews, monitor news mentions
Automate 5-10 hours/week of manual web research
Content Monitoring & Alerts
Track website changes, new content, price updates
Example
Monitor competitor blog for new posts, track stock availability, watch for pricing changes
Stay informed without manual checking, never miss important updates
Data Extraction & Aggregation
Extract structured data from multiple websites
Example
Compile product listings from 10 e-commerce sites, aggregate job postings, collect real estate data
Build datasets 100x faster than manual copying
API-less Integration
Interact with services that don't offer APIs
Example
Check form submissions, validate website functionality, test user flows
Automate interactions with any website, even without API
Implementation Guide▌
Prerequisites
- ›Claude Desktop or Cursor with MCP support
- ›Understanding of web scraping ethics and robots.txt
- ›Rate limiting awareness to avoid overwhelming target sites
- ›Knowledge of legal restrictions on data collection
Time Estimate
20-40 minutes including configuration and testing
Installation Steps
- 1.Install web automation MCP server via npm or pip
- 2.Configure allowed domains and rate limits in MCP config
- 3.Test with simple fetch: 'Get content from example.com'
- 4.Progress to extraction: 'Extract all product prices from this page'
- 5.Set up monitoring: 'Check this URL daily for changes'
- 6.Parse structured data: 'Create CSV from this table'
- 7.Respect robots.txt and rate limits always
Troubleshooting
- ⚠403 Forbidden: Website blocks bots—respect their wishes, use official API instead
- ⚠Rate limit errors: Slow down requests, add delays between fetches
- ⚠Stale data: Target site changed HTML structure—update selectors
- ⚠Timeout errors: Site is slow or blocking—increase timeout, try different user agent
- ⚠JavaScript-rendered content: Use headless browser MCP servers for dynamic sites
Best Practices▌
✓ Do
- +Check robots.txt and respect crawl rules
- +Rate limit requests: 1-2 requests/second maximum
- +Use official APIs when available instead of scraping
- +Identify your bot with descriptive user agent
- +Cache results to minimize repeated requests
- +Handle errors gracefully with retries and fallbacks
- +Validate extracted data for accuracy
✗ Don't
- −Don't scrape sites that explicitly forbid it (robots.txt, ToS)
- −Don't overwhelm servers with rapid requests—use rate limiting
- −Don't scrape personal data without consent and legal basis
- −Don't ignore copyright on extracted content
- −Don't assume HTML structure is stable—handle changes
- −Don't use scraped data for commercial purposes without permission
💡 Pro Tips
- ★Use CSS selectors or XPath for robust data extraction
- ★Set up monitoring alerts for extraction failures (structure changed)
- ★Implement exponential backoff for retries on failures
- ★Store raw HTML for reprocessing if extraction logic changes
- ★Combine with data analysis tools for insights from extracted data
- ★Consider using official APIs or RSS feeds as more stable alternatives
Technical Details▌
Architecture
MCP server handles HTTP requests, HTML parsing, JavaScript rendering (if headless browser), and returns structured data to Claude.
Protocols
- HTTP/HTTPS
- WebSocket (for real-time sites)
- Puppeteer/Playwright (for JavaScript sites)
Compatibility
- Static HTML sites
- JavaScript-rendered SPAs (with headless browser)
- REST APIs
- GraphQL endpoints
When to Use This▌
✓ Use When
Use for research automation, content monitoring, data aggregation from multiple sources, and when official APIs don't exist. Best for read-only information gathering.
✗ Avoid When
Avoid for sites with APIs (use API instead), sites that explicitly forbid scraping, when data is copyrighted, or for login-required content without proper authorization.
Integration▌
- →Scheduled monitoring with change detection
- →Multi-source data aggregation pipelines
- →Fallback to web scraping when API rate limits hit
- →Headless browser for JavaScript-heavy sites
Discussion
Product Hunt–style comments (not star reviews)- No comments yet — start the thread.
List & Promote Your MCP Server
Share your MCP server with the developer community
Ratings
4.6★★★★★54 reviews- ★★★★★Aditi Wang· Dec 12, 2024
We wired Web Fetch into a staging workspace; the listing’s GitHub and npm pointers saved time versus hunting across READMEs.
- ★★★★★Aditi Li· Dec 4, 2024
Web Fetch reduced integration guesswork — categories and install configs on the listing matched the upstream repo.
- ★★★★★Hana Desai· Nov 23, 2024
I recommend Web Fetch for teams standardizing on MCP; the explainx.ai page compares cleanly with sibling servers.
- ★★★★★Yash Thakker· Nov 15, 2024
Web Fetch is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.
- ★★★★★Aanya Jackson· Nov 3, 2024
According to our notes, Web Fetch benefits from clear Model Context Protocol framing — fewer ambiguous “AI plugin” claims.
- ★★★★★Aanya Park· Oct 22, 2024
Web Fetch has been reliable for tool-calling workflows; the MCP profile page is a good permalink for internal docs.
- ★★★★★Hana Thompson· Oct 14, 2024
Strong directory entry: Web Fetch surfaces stars and publisher context so we could sanity-check maintenance before adopting.
- ★★★★★Dhruvi Jain· Oct 6, 2024
We evaluated Web Fetch against two servers with overlapping tools; this profile had the clearer scope statement.
- ★★★★★Min Johnson· Sep 25, 2024
According to our notes, Web Fetch benefits from clear Model Context Protocol framing — fewer ambiguous “AI plugin” claims.
- ★★★★★Kwame Chen· Sep 25, 2024
Web Fetch is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.
showing 1-10 of 54