search-web

Oxylabs Web Scraping

oxylabs

by oxylabs

Integrate with Oxylabs web scraping for real-time data extraction, cleaning, and analysis. Easily scrape any website for

Integrates with Oxylabs web scraping services to extract, clean, and structure web content for real-time data analysis and monitoring workflows.

github stars

87

0 commentsdiscussion

Both formats append explainx.ai attribution and the canonical URL for this MCP server listing.

Bypasses anti-scraping measuresJavaScript rendering supportGlobal geo-location coverage

best for

  • / AI developers building data-driven applications
  • / Market research and competitive analysis
  • / E-commerce price monitoring
  • / SEO and search result tracking

capabilities

  • / Scrape any website URL with anti-bot bypass
  • / Extract Google search results
  • / Scrape Amazon product and search pages
  • / Render JavaScript-heavy websites
  • / Access geo-restricted content from 195+ countries
  • / Format scraped data for AI consumption

what it does

Connects AI models to Oxylabs web scraping API to extract and structure data from any website, including JavaScript-heavy pages and geo-restricted content.

about

Oxylabs Web Scraping is an official MCP server published by oxylabs that provides AI assistants with tools and capabilities via the Model Context Protocol. Integrate with Oxylabs web scraping for real-time data extraction, cleaning, and analysis. Easily scrape any website for It is categorized under search web.

how to install

You can install Oxylabs Web Scraping in your AI client of choice. Use the install panel on this page to get one-click setup for Cursor, Claude Desktop, VS Code, and other MCP-compatible clients. This server runs locally on your machine via the stdio transport.

license

MIT

Oxylabs Web Scraping is released under the MIT license. This is a permissive open-source license, meaning you can freely use, modify, and distribute the software.

readme

Oxylabs + MCP

Oxylabs MCP Server

The missing link between AI models and the real‑world web: one API that delivers clean, structured data from any site.

[![smithery badge](https://smithery.ai/badge/@oxylabs/oxylabs-mcp)](https://smithery.ai/server/@oxylabs/oxylabs-mcp) [![pypi package](https://img.shields.io/pypi/v/oxylabs-mcp?color=%2334D058&label=pypi%20package)](https://pypi.org/project/oxylabs-mcp/) [![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5?style=flat)](https://discord.gg/Pds3gBmKMH) [![Licence](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) [![Verified on MseeP](https://mseep.ai/badge.svg)](https://mseep.ai/app/f6a9c0bc-83a6-4f78-89d9-f2cec4ece98d) ![Coverage badge](https://raw.githubusercontent.com/oxylabs/oxylabs-mcp/coverage/coverage-badge.svg)
Oxylabs Server MCP server
--- ## 📖 Overview The Oxylabs MCP server provides a bridge between AI models and the web. It enables them to scrape any URL, render JavaScript-heavy pages, extract and format content for AI use, bypass anti-scraping measures, and access geo-restricted web data from 195+ countries. ## 🛠️ MCP Tools Oxylabs MCP provides two sets of tools that can be used together or independently: ### Oxylabs Web Scraper API Tools 1. **universal_scraper**: Uses Oxylabs Web Scraper API for general website scraping; 2. **google_search_scraper**: Uses Oxylabs Web Scraper API to extract results from Google Search; 3. **amazon_search_scraper**: Uses Oxylabs Web Scraper API to scrape Amazon search result pages; 4. **amazon_product_scraper**: Uses Oxylabs Web Scraper API to extract data from individual Amazon product pages. ### Oxylabs AI Studio Tools 5. **ai_scraper**: Scrape content from any URL in JSON or Markdown format with AI-powered data extraction; 6. **ai_crawler**: Based on a prompt, crawls a website and collects data in Markdown or JSON format across multiple pages; 7. **ai_browser_agent**: Based on prompt, controls a browser and returns data in Markdown, JSON, HTML, or screenshot formats; 8. **ai_search**: Search the web for URLs and their contents with AI-powered content extraction. ## ✅ Prerequisites Before you begin, make sure you have **at least one** of the following: - **Oxylabs Web Scraper API Account**: Obtain your username and password from [Oxylabs](https://dashboard.oxylabs.io/) (1-week free trial available); - **Oxylabs AI Studio API Key**: Obtain your API key from [Oxylabs AI Studio](https://aistudio.oxylabs.io/settings/api-key). (1000 credits free). ## 📦 Configuration ### Environment variables Oxylabs MCP server supports the following environment variables: | Name | Description | Default | |----------------------------|-----------------------------------------------|---------| | `OXYLABS_USERNAME` | Your Oxylabs Web Scraper API username | | | `OXYLABS_PASSWORD` | Your Oxylabs Web Scraper API password | | | `OXYLABS_AI_STUDIO_API_KEY`| Your Oxylabs AI Studio API key | | | `LOG_LEVEL` | Log level for the logs returned to the client | `INFO` | Based on provided credentials, the server will automatically expose the corresponding tools: - If only `OXYLABS_USERNAME` and `OXYLABS_PASSWORD` are provided, the server will expose the Web Scraper API tools; - If only `OXYLABS_AI_STUDIO_API_KEY` is provided, the server will expose the AI Studio tools; - If both `OXYLABS_USERNAME` and `OXYLABS_PASSWORD` and `OXYLABS_AI_STUDIO_API_KEY` are provided, the server will expose all tools. ❗❗❗ **Important note: if you don't have Web Scraper API or Oxylabs AI studio credentials, delete the corresponding environment variables placeholders. Leaving placeholder values will result in exposed tools that do not work.** ### Configure with uvx - Install the uvx package manager: ```bash # macOS and Linux curl -LsSf https://astral.sh/uv/install.sh | sh ``` OR: ```bash # Windows powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" ``` - Use the following config: ```json { "mcpServers": { "oxylabs": { "command": "uvx", "args": ["oxylabs-mcp"], "env": { "OXYLABS_USERNAME": "OXYLABS_USERNAME", "OXYLABS_PASSWORD": "OXYLABS_PASSWORD", "OXYLABS_AI_STUDIO_API_KEY": "OXYLABS_AI_STUDIO_API_KEY" } } } } ``` ### Configure with uv - Install the uv package manager: ```bash # macOS and Linux curl -LsSf https://astral.sh/uv/install.sh | sh ``` OR: ```bash # Windows powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" ``` - Use the following config: ```json { "mcpServers": { "oxylabs": { "command": "uv", "args": [ "--directory", "//oxylabs-mcp", "run", "oxylabs-mcp" ], "env": { "OXYLABS_USERNAME": "OXYLABS_USERNAME", "OXYLABS_PASSWORD": "OXYLABS_PASSWORD", "OXYLABS_AI_STUDIO_API_KEY": "OXYLABS_AI_STUDIO_API_KEY" } } } } ``` ### Configure with Smithery Oauth2 - Go to https://smithery.ai/server/@oxylabs/oxylabs-mcp; - Click _Auto_ to install the Oxylabs MCP configuration for the respective client; - OR use the following config: ```json { "mcpServers": { "oxylabs": { "url": "https://server.smithery.ai/@oxylabs/oxylabs-mcp/mcp" } } } ``` - Follow the instructions to authenticate Oxylabs MCP with Oauth2 flow ### Configure with Smithery query parameters In case your client does not support the Oauth2 authentication, you can pass the Oxylabs authentication parameters directly in url ```json { "mcpServers": { "oxylabs": { "url": "https://server.smithery.ai/@oxylabs/oxylabs-mcp/mcp?oxylabsUsername=OXYLABS_USERNAME&oxylabsPassword=OXYLABS_PASSWORD&oxylabsAiStudioApiKey=OXYLABS_AI_STUDIO_API_KEY" } } } ``` ### Manual Setup with Claude Desktop Navigate to **Claude → Settings → Developer → Edit Config** and add one of the configurations above to the `claude_desktop_config.json` file. ### Manual Setup with Cursor AI Navigate to **Cursor → Settings → Cursor Settings → MCP**. Click **Add new global MCP server** and add one of the configurations above. ## 📝 Logging Server provides additional information about the tool calls in `notification/message` events ```json { "method": "notifications/message", "params": { "level": "info", "data": "Create job with params: {"url": "https://ip.oxylabs.io"}" } } ``` ```json { "method": "notifications/message", "params": { "level": "info", "data": "Job info: job_id=7333113830223918081 job_status=done" } } ``` ```json { "method": "notifications/message", "params": { "level": "error", "data": "Error: request to Oxylabs API failed" } } ``` --- ## 🛡️ License Distributed under the MIT License – see [LICENSE](LICENSE) for details. --- ## About Oxylabs Established in 2015, Oxylabs is a market-leading web intelligence collection platform, driven by the highest business, ethics, and compliance standards, enabling companies worldwide to unlock data-driven insights. [![image](https://oxylabs.io/images/og-image.png)](https://oxylabs.io/)
Made with ☕ by Oxylabs. Feel free to give us a ⭐ if MCP saved you a weekend.
## ✨ Key Features
Scrape content from any site
- Extract data from any URL, including complex single-page applications - Fully render dynamic websites using headless browser support - Choose full JavaScript rendering, HTML-only, or none - Emulate Mobile and Desktop viewports for realistic rendering
Automatically get AI-ready data
- Automatically clean and convert HTML to Markdown for improved readability - Use automated parsers for popular targets like Google, Amazon, and more
Bypass blocks & geo-restrictions
- Bypass sophisticated bot protection systems with high success rate - Reliably scrape even the most complex websites - Get automatically rotating IPs from a proxy pool covering 195+ countries
Flexible setup & cross-platform support
- Set rendering and parsing options if needed - Feed data directly into AI models or analytics tools - Works on macOS, Windows, and Linux
Built-in error handling and request management
- Comprehensive error handling and reporting - Smart rate limiting and request management
--- ## Why Oxylabs MCP?  🕸️ ➜ 📦 ➜ 🤖 Imagine telling your LLM *"Summarise the latest Hacker News discussion about GPT‑5"* – and it simply answers. MCP (Multi‑Client Proxy) makes that happen by doing the boring parts for you: | What Oxylabs MCP does | Why it matters to you | |-------------------------------------------------------------------|------------------------------------------| | **Bypasses anti‑bot walls** with the Oxylabs global proxy network | Keeps you unblocked and anonymous | | **Renders JavaScript** in headless Chrome | Single‑page apps, sorted | | **Cleans H ---

FAQ

What is the Oxylabs Web Scraping MCP server?
Oxylabs Web Scraping is a Model Context Protocol (MCP) server profile on explainx.ai. MCP lets AI hosts (e.g. Claude Desktop, Cursor) call tools and resources through a standard interface; this page summarizes categories, install hints, and community ratings.
How do MCP servers relate to agent skills?
Skills are reusable instruction packages (often SKILL.md); MCP servers expose live capabilities. Teams frequently combine both—skills for workflows, MCP for APIs and data. See explainx.ai/skills and explainx.ai/mcp-servers for parallel directories.
How are reviews shown for Oxylabs Web Scraping?
This profile displays 69 aggregated ratings (sample rows for discoverability plus signed-in user reviews). Average score is about 4.5 out of 5—verify behavior in your own environment before production use.

Use Cases

Web Research & Information Gathering

Fetch and extract information from websites automatically

Example

Research competitor pricing, scrape product reviews, monitor news mentions

Automate 5-10 hours/week of manual web research

Content Monitoring & Alerts

Track website changes, new content, price updates

Example

Monitor competitor blog for new posts, track stock availability, watch for pricing changes

Stay informed without manual checking, never miss important updates

Data Extraction & Aggregation

Extract structured data from multiple websites

Example

Compile product listings from 10 e-commerce sites, aggregate job postings, collect real estate data

Build datasets 100x faster than manual copying

API-less Integration

Interact with services that don't offer APIs

Example

Check form submissions, validate website functionality, test user flows

Automate interactions with any website, even without API

Implementation Guide

Prerequisites

  • Claude Desktop or Cursor with MCP support
  • Understanding of web scraping ethics and robots.txt
  • Rate limiting awareness to avoid overwhelming target sites
  • Knowledge of legal restrictions on data collection

Time Estimate

20-40 minutes including configuration and testing

Installation Steps

  1. 1.Install web automation MCP server via npm or pip
  2. 2.Configure allowed domains and rate limits in MCP config
  3. 3.Test with simple fetch: 'Get content from example.com'
  4. 4.Progress to extraction: 'Extract all product prices from this page'
  5. 5.Set up monitoring: 'Check this URL daily for changes'
  6. 6.Parse structured data: 'Create CSV from this table'
  7. 7.Respect robots.txt and rate limits always

Troubleshooting

  • 403 Forbidden: Website blocks bots—respect their wishes, use official API instead
  • Rate limit errors: Slow down requests, add delays between fetches
  • Stale data: Target site changed HTML structure—update selectors
  • Timeout errors: Site is slow or blocking—increase timeout, try different user agent
  • JavaScript-rendered content: Use headless browser MCP servers for dynamic sites

Best Practices

✓ Do

  • +Check robots.txt and respect crawl rules
  • +Rate limit requests: 1-2 requests/second maximum
  • +Use official APIs when available instead of scraping
  • +Identify your bot with descriptive user agent
  • +Cache results to minimize repeated requests
  • +Handle errors gracefully with retries and fallbacks
  • +Validate extracted data for accuracy

✗ Don't

  • Don't scrape sites that explicitly forbid it (robots.txt, ToS)
  • Don't overwhelm servers with rapid requests—use rate limiting
  • Don't scrape personal data without consent and legal basis
  • Don't ignore copyright on extracted content
  • Don't assume HTML structure is stable—handle changes
  • Don't use scraped data for commercial purposes without permission

💡 Pro Tips

  • Use CSS selectors or XPath for robust data extraction
  • Set up monitoring alerts for extraction failures (structure changed)
  • Implement exponential backoff for retries on failures
  • Store raw HTML for reprocessing if extraction logic changes
  • Combine with data analysis tools for insights from extracted data
  • Consider using official APIs or RSS feeds as more stable alternatives

Technical Details

Architecture

MCP server handles HTTP requests, HTML parsing, JavaScript rendering (if headless browser), and returns structured data to Claude.

Protocols

  • HTTP/HTTPS
  • WebSocket (for real-time sites)
  • Puppeteer/Playwright (for JavaScript sites)

Compatibility

  • Static HTML sites
  • JavaScript-rendered SPAs (with headless browser)
  • REST APIs
  • GraphQL endpoints

When to Use This

✓ Use When

Use for research automation, content monitoring, data aggregation from multiple sources, and when official APIs don't exist. Best for read-only information gathering.

✗ Avoid When

Avoid for sites with APIs (use API instead), sites that explicitly forbid scraping, when data is copyrighted, or for login-required content without proper authorization.

Integration

  • Scheduled monitoring with change detection
  • Multi-source data aggregation pipelines
  • Fallback to web scraping when API rate limits hit
  • Headless browser for JavaScript-heavy sites

Discussion

Product Hunt–style comments (not star reviews)
  • No comments yet — start the thread.

List & Promote Your MCP Server

Share your MCP server with the developer community

GET_STARTED →
MCP server reviews

Ratings

4.569 reviews
  • Zara White· Dec 24, 2024

    Strong directory entry: Oxylabs Web Scraping surfaces stars and publisher context so we could sanity-check maintenance before adopting.

  • Isabella Abebe· Dec 12, 2024

    I recommend Oxylabs Web Scraping for teams standardizing on MCP; the explainx.ai page compares cleanly with sibling servers.

  • James Verma· Dec 8, 2024

    I recommend Oxylabs Web Scraping for teams standardizing on MCP; the explainx.ai page compares cleanly with sibling servers.

  • Nikhil Johnson· Nov 27, 2024

    We evaluated Oxylabs Web Scraping against two servers with overlapping tools; this profile had the clearer scope statement.

  • Hana Mehta· Nov 15, 2024

    Oxylabs Web Scraping is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.

  • Kaira Chawla· Nov 3, 2024

    Oxylabs Web Scraping reduced integration guesswork — categories and install configs on the listing matched the upstream repo.

  • Neel Gupta· Nov 3, 2024

    Useful MCP listing: Oxylabs Web Scraping is the kind of server we cite when onboarding engineers to host + tool permissions.

  • Hana Srinivasan· Nov 3, 2024

    We evaluated Oxylabs Web Scraping against two servers with overlapping tools; this profile had the clearer scope statement.

  • James Tandon· Oct 22, 2024

    Useful MCP listing: Oxylabs Web Scraping is the kind of server we cite when onboarding engineers to host + tool permissions.

  • Zara Jackson· Oct 22, 2024

    Oxylabs Web Scraping reduced integration guesswork — categories and install configs on the listing matched the upstream repo.

showing 1-10 of 69

1 / 7