search-web

ScrAPI

deventerprisesoftware

by deventerprisesoftware

ScrAPI lets you scrape any website with ease, bypassing bot detection and captchas using residential proxies for reliabl

Enables web scraping from sites with bot detection, captchas, or geolocation restrictions through residential proxies and automated captcha solving for content extraction in HTML or Markdown formats.

github stars

18

0 commentsdiscussion

Both formats append explainx.ai attribution and the canonical URL for this MCP server listing.

Residential proxy networkAutomatic captcha solvingBypasses geolocation restrictions

best for

  • / Data extraction from protected websites
  • / Scraping e-commerce sites with anti-bot measures
  • / Accessing content blocked by region
  • / Research requiring bot-protected data

capabilities

  • / Scrape websites with bot detection
  • / Bypass captchas automatically
  • / Access geo-restricted content
  • / Execute browser commands before scraping
  • / Extract content as HTML or Markdown
  • / Handle complex JavaScript sites

what it does

Scrapes websites that block regular bots by using residential proxies and automated captcha solving. Returns content in HTML or Markdown format.

about

ScrAPI is an official MCP server published by deventerprisesoftware that provides AI assistants with tools and capabilities via the Model Context Protocol. ScrAPI lets you scrape any website with ease, bypassing bot detection and captchas using residential proxies for reliabl It is categorized under search web.

how to install

You can install ScrAPI in your AI client of choice. Use the install panel on this page to get one-click setup for Cursor, Claude Desktop, VS Code, and other MCP-compatible clients. This server runs locally on your machine via the stdio transport.

license

MIT

ScrAPI is released under the MIT license. This is a permissive open-source license, meaning you can freely use, modify, and distribute the software.

readme

ScrAPI logo

ScrAPI MCP Server

License: MIT NPM Downloads Docker Pulls smithery badge

MCP server for using ScrAPI to scrape web pages.

ScrAPI is your ultimate web scraping solution, offering powerful, reliable, and easy-to-use features to extract data from any website effortlessly.

<a href="https://glama.ai/mcp/servers/@DevEnterpriseSoftware/scrapi-mcp"> <img width="380" height="200" src="https://glama.ai/mcp/servers/@DevEnterpriseSoftware/scrapi-mcp/badge" alt="ScrAPI Server MCP server" /> </a>

Tools

  1. scrape_url_html

    • Use a URL to scrape a website using the ScrAPI service and retrieve the result as HTML. Use this for scraping website content that is difficult to access because of bot detection, captchas or even geolocation restrictions. The result will be in HTML which is preferable if advanced parsing is required.
    • Inputs:
      • url (string, required): The URL to scrape
      • browserCommands (string, optional): JSON array of browser commands to execute before scraping
    • Returns: HTML content of the URL
  2. scrape_url_markdown

    • Use a URL to scrape a website using the ScrAPI service and retrieve the result as Markdown. Use this for scraping website content that is difficult to access because of bot detection, captchas or even geolocation restrictions. The result will be in Markdown which is preferable if the text content of the webpage is important and not the structural information of the page.
    • Inputs:
      • url (string, required): The URL to scrape
      • browserCommands (string, optional): JSON array of browser commands to execute before scraping
    • Returns: Markdown content of the URL

Browser Commands

Both tools support optional browser commands that allow you to interact with the page before scraping. This is useful for:

  • Clicking buttons (e.g., "Accept Cookies", "Load More")
  • Filling out forms
  • Selecting dropdown options
  • Scrolling to load dynamic content
  • Waiting for elements to appear
  • Executing custom JavaScript

Available Commands

Commands are provided as a JSON array string. All commands are executed with human-like behavior (random mouse movements, variable typing speed, etc.):

CommandFormatDescription
Click{"click": "#buttonId"}Click an element using CSS selector
Input{"input": {"input[name='email']": "value"}}Fill an input field
Select{"select": {"select[name='country']": "USA"}}Select from dropdown (by value or text)
Scroll{"scroll": 1000}Scroll down by pixels (negative values scroll up)
Wait{"wait": 5000}Wait for milliseconds (max 15000)
WaitFor{"waitfor": "#elementId"}Wait for element to appear in DOM
JavaScript{"javascript": "console.log('test')"}Execute custom JavaScript code

Example Usage

[
  {"click": "#accept-cookies"},
  {"wait": 2000},
  {"input": {"input[name='search']": "web scraping"}},
  {"click": "button[type='submit']"},
  {"waitfor": "#results"},
  {"scroll": 500}
]

Finding CSS Selectors

Need help finding CSS selectors? Try the Rayrun browser extension to easily select elements and generate selectors.

For more details, see the Browser Commands documentation.

Setup

API Key (optional)

Optionally get an API key from the ScrAPI website.

Without an API key you will be limited to one concurrent call and twenty free calls per day with minimal queuing capabilities.

Cloud Server

The ScrAPI MCP Server is also available in the cloud over SSE at https://api.scrapi.tech/mcp/sse and streamable HTTP at https://api.scrapi.tech/mcp

Cloud MCP servers are not widely supported yet but you can access this directly from your own custom clients or use MCP Inspector to test it. There is currently no facility to pass through your API key when connecting to the cloud MCP server.

MCP-Inspector

Usage with Claude Desktop

Add the following to your claude_desktop_config.json:

Docker

{
  "mcpServers": {
    "ScrAPI": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "SCRAPI_API_KEY",
        "deventerprisesoftware/scrapi-mcp"
      ],
      "env": {
        "SCRAPI_API_KEY": "<YOUR_API_KEY>"
      }
    }
  }
}

NPX

{
  "mcpServers": {
    "ScrAPI": {
      "command": "npx",
      "args": [
        "-y",
        "@deventerprisesoftware/scrapi-mcp"
      ],
      "env": {
        "SCRAPI_API_KEY": "<YOUR_API_KEY>"
      }
    }
  }
}

Claude-Desktop

Build

Docker build:

docker build -t deventerprisesoftware/scrapi-mcp -f Dockerfile .

License

This MCP server is licensed under the MIT License. This means you are free to use, modify, and distribute the software, subject to the terms and conditions of the MIT License. For more details, please see the LICENSE file in the project repository.

FAQ

What is the ScrAPI MCP server?
ScrAPI is a Model Context Protocol (MCP) server profile on explainx.ai. MCP lets AI hosts (e.g. Claude Desktop, Cursor) call tools and resources through a standard interface; this page summarizes categories, install hints, and community ratings.
How do MCP servers relate to agent skills?
Skills are reusable instruction packages (often SKILL.md); MCP servers expose live capabilities. Teams frequently combine both—skills for workflows, MCP for APIs and data. See explainx.ai/skills and explainx.ai/mcp-servers for parallel directories.
How are reviews shown for ScrAPI?
This profile displays 30 aggregated ratings (sample rows for discoverability plus signed-in user reviews). Average score is about 4.5 out of 5—verify behavior in your own environment before production use.

Use Cases

Web Research & Information Gathering

Fetch and extract information from websites automatically

Example

Research competitor pricing, scrape product reviews, monitor news mentions

Automate 5-10 hours/week of manual web research

Content Monitoring & Alerts

Track website changes, new content, price updates

Example

Monitor competitor blog for new posts, track stock availability, watch for pricing changes

Stay informed without manual checking, never miss important updates

Data Extraction & Aggregation

Extract structured data from multiple websites

Example

Compile product listings from 10 e-commerce sites, aggregate job postings, collect real estate data

Build datasets 100x faster than manual copying

API-less Integration

Interact with services that don't offer APIs

Example

Check form submissions, validate website functionality, test user flows

Automate interactions with any website, even without API

Implementation Guide

Prerequisites

  • Claude Desktop or Cursor with MCP support
  • Understanding of web scraping ethics and robots.txt
  • Rate limiting awareness to avoid overwhelming target sites
  • Knowledge of legal restrictions on data collection

Time Estimate

20-40 minutes including configuration and testing

Installation Steps

  1. 1.Install web automation MCP server via npm or pip
  2. 2.Configure allowed domains and rate limits in MCP config
  3. 3.Test with simple fetch: 'Get content from example.com'
  4. 4.Progress to extraction: 'Extract all product prices from this page'
  5. 5.Set up monitoring: 'Check this URL daily for changes'
  6. 6.Parse structured data: 'Create CSV from this table'
  7. 7.Respect robots.txt and rate limits always

Troubleshooting

  • 403 Forbidden: Website blocks bots—respect their wishes, use official API instead
  • Rate limit errors: Slow down requests, add delays between fetches
  • Stale data: Target site changed HTML structure—update selectors
  • Timeout errors: Site is slow or blocking—increase timeout, try different user agent
  • JavaScript-rendered content: Use headless browser MCP servers for dynamic sites

Best Practices

✓ Do

  • +Check robots.txt and respect crawl rules
  • +Rate limit requests: 1-2 requests/second maximum
  • +Use official APIs when available instead of scraping
  • +Identify your bot with descriptive user agent
  • +Cache results to minimize repeated requests
  • +Handle errors gracefully with retries and fallbacks
  • +Validate extracted data for accuracy

✗ Don't

  • Don't scrape sites that explicitly forbid it (robots.txt, ToS)
  • Don't overwhelm servers with rapid requests—use rate limiting
  • Don't scrape personal data without consent and legal basis
  • Don't ignore copyright on extracted content
  • Don't assume HTML structure is stable—handle changes
  • Don't use scraped data for commercial purposes without permission

💡 Pro Tips

  • Use CSS selectors or XPath for robust data extraction
  • Set up monitoring alerts for extraction failures (structure changed)
  • Implement exponential backoff for retries on failures
  • Store raw HTML for reprocessing if extraction logic changes
  • Combine with data analysis tools for insights from extracted data
  • Consider using official APIs or RSS feeds as more stable alternatives

Technical Details

Architecture

MCP server handles HTTP requests, HTML parsing, JavaScript rendering (if headless browser), and returns structured data to Claude.

Protocols

  • HTTP/HTTPS
  • WebSocket (for real-time sites)
  • Puppeteer/Playwright (for JavaScript sites)

Compatibility

  • Static HTML sites
  • JavaScript-rendered SPAs (with headless browser)
  • REST APIs
  • GraphQL endpoints

When to Use This

✓ Use When

Use for research automation, content monitoring, data aggregation from multiple sources, and when official APIs don't exist. Best for read-only information gathering.

✗ Avoid When

Avoid for sites with APIs (use API instead), sites that explicitly forbid scraping, when data is copyrighted, or for login-required content without proper authorization.

Integration

  • Scheduled monitoring with change detection
  • Multi-source data aggregation pipelines
  • Fallback to web scraping when API rate limits hit
  • Headless browser for JavaScript-heavy sites

Discussion

Product Hunt–style comments (not star reviews)
  • No comments yet — start the thread.

List & Promote Your MCP Server

Share your MCP server with the developer community

GET_STARTED →
MCP server reviews

Ratings

4.530 reviews
  • Michael Flores· Dec 16, 2024

    According to our notes, ScrAPI benefits from clear Model Context Protocol framing — fewer ambiguous “AI plugin” claims.

  • Aanya Jackson· Dec 12, 2024

    ScrAPI reduced integration guesswork — categories and install configs on the listing matched the upstream repo.

  • Aanya Malhotra· Dec 12, 2024

    I recommend ScrAPI for teams standardizing on MCP; the explainx.ai page compares cleanly with sibling servers.

  • Ganesh Mohane· Dec 4, 2024

    ScrAPI has been reliable for tool-calling workflows; the MCP profile page is a good permalink for internal docs.

  • Sakshi Patil· Nov 23, 2024

    ScrAPI reduced integration guesswork — categories and install configs on the listing matched the upstream repo.

  • Michael Lopez· Nov 3, 2024

    ScrAPI has been reliable for tool-calling workflows; the MCP profile page is a good permalink for internal docs.

  • Aanya Johnson· Nov 3, 2024

    Strong directory entry: ScrAPI surfaces stars and publisher context so we could sanity-check maintenance before adopting.

  • Noah Okafor· Oct 22, 2024

    ScrAPI is a well-scoped MCP server in the explainx.ai directory — install snippets and categories matched our Claude Code setup.

  • Tariq Patel· Oct 22, 2024

    Useful MCP listing: ScrAPI is the kind of server we cite when onboarding engineers to host + tool permissions.

  • Chaitanya Patil· Oct 14, 2024

    We wired ScrAPI into a staging workspace; the listing’s GitHub and npm pointers saved time versus hunting across READMEs.

showing 1-10 of 30

1 / 3