search-web

mcp-server-rag-web-browser

apify

by apify

MCP server implementing the RAG Web Browser Actor — a web browsing/search tool for LLMs and RAG pipelines, enabling web-

Implementation of an MCP server for the RAG Web Browser Actor. This Actor serves as a web browser for large language models (LLMs) and RAG pipelines, similar to a web search in ChatGPT.

github stars

198

0 commentsdiscussion

Both formats append explainx.ai attribution and the canonical URL for this MCP server listing.

best for

  • / General purpose MCP workflows

capabilities

    what it does

    Implementation of an MCP server for the RAG Web Browser Actor. This Actor serves as a web browser for large language models (LLMs) and RAG pipelines, similar to a web search in ChatGPT.

    about

    mcp-server-rag-web-browser is an official MCP server published by apify that provides AI assistants with tools and capabilities via the Model Context Protocol. MCP server implementing the RAG Web Browser Actor — a web browsing/search tool for LLMs and RAG pipelines, enabling web- It is categorized under search web.

    how to install

    You can install mcp-server-rag-web-browser in your AI client of choice. Use the install panel on this page to get one-click setup for Cursor, Claude Desktop, VS Code, and other MCP-compatible clients. This server runs locally on your machine via the stdio transport.

    license

    Apache-2.0

    mcp-server-rag-web-browser is released under the Apache-2.0 license. This is a permissive open-source license, meaning you can freely use, modify, and distribute the software.

    readme

    MCP Server for the RAG Web Browser Actor 🌐

    Implementation of an MCP server for the RAG Web Browser Actor. This Actor serves as a web browser for large language models (LLMs) and RAG pipelines, similar to a web search in ChatGPT.

    This MCP server is deprecated in favor of mcp.apify.com

    For the same functionality and much more, please use one of these alternatives:

    🚀 Recommended: use mcp.apify.com

    The easiest way to get the same web browsing capabilities is to use mcp.apify.com with default settings.

    Benefits:

    • ✅ No local setup required
    • ✅ Always up-to-date
    • ✅ Access to 6,000+ Apify Actors (including RAG Web Browser)
    • ✅ OAuth support for easy connection
    • ✅ Dynamic tool discovery

    Quick Setup:

    1. Go to https://mcp.apify.com
    2. Authorize the client (Claude, VS Code, etc.)
    3. Copy the generated MCP server configuration (or use OAuth flow if supported)
    4. Start using browsing & other tools immediately

    🌐 Alternative: direct RAG Web Browser integration

    You can also call the RAG Web Browser Actor directly via its HTTP/SSE interface.

    Benefits:

    • ✅ Direct integration without mcp.apify.com
    • ✅ Real-time streaming via Server-Sent Events
    • ✅ Full control over the integration
    • ✅ No additional dependencies

    Docs: Actor Documentation


    🎯 What does this MCP server do?

    This server is specifically designed to provide fast responses to AI agents and LLMs, allowing them to interact with the web and extract information from web pages. It runs locally and communicates with the RAG Web Browser Actor in Standby mode, sending search queries and receiving extracted web content in response.

    • Web Search: Query Google Search, scrape top N URLs, and return cleaned content as Markdown
    • Single URL Fetching: Fetch a specific URL and return its content as Markdown
    • Local MCP Integration: Standard input/output (stdio) communication with AI clients

    🧱 Components

    Tools

    • name: search description: Query Google Search OR fetch a direct URL and return cleaned page contents. arguments:
      • query (string, required): Search keywords or a full URL. Advanced Google operators supported.
      • maxResults (number, optional, default: 1): Max organic results to fetch (ignored when query is a URL).
      • scrapingTool (string, optional, default: raw-http): One of browser-playwright | raw-http.
        • raw-http: Fast (no JS execution) – good for static pages.
        • browser-playwright: Handles JS-heavy sites – slower, more robust.
      • outputFormats (array of strings, optional, default: [markdown]): One or more of text, markdown, html.
      • requestTimeoutSecs (number, optional, default: 40, min 1 max 300): Total server-side AND client wait budget. A local abort is enforced.

    🔄 Migration Guide

    From Local MCP Server to mcp.apify.com

    Before (Deprecated local server):

    {
      "mcpServers": {
        "rag-web-browser": {
          "command": "npx",
          "args": ["@apify/mcp-server-rag-web-browser"],
          "env": {
            "APIFY_TOKEN": "your-apify-api-token"
          }
        }
      }
    }
    

    After (Recommended Apify server):

    {
      "mcpServers": {
        "apify": {
          "command": "npx",
          "args": ["@apify/actors-mcp-server"],
          "env": {
            "APIFY_TOKEN": "your-apify-api-token"
          }
        }
      }
    }
    

    Or use the hosted endpoint: https://mcp.apify.com (when your client supports HTTP transport / remote MCP).

    MCP clients

    🛠️ Development

    Prerequisites

    • Node.js (v18 or higher)
    • Apify API Token (APIFY_TOKEN)

    Clone & install:

    git clone https://github.com/apify/mcp-server-rag-web-browser.git
    cd mcp-server-rag-web-browser
    npm install
    

    Build

    npm install
    npm run build
    

    Debugging

    Since MCP servers operate over standard input/output (stdio), debugging can be challenging. For the best debugging experience, use the MCP Inspector.

    You can launch the MCP Inspector via npm with this command:

    export APIFY_TOKEN=your-apify-api-token
    npx @modelcontextprotocol/inspector node dist/index.js
    

    Upon launching, the Inspector will display a URL that you can access in your browser to begin debugging.

    📖 Learn more

    This repository is maintained for archival purposes only. Please use the recommended alternatives above for active development.

    FAQ

    What is the mcp-server-rag-web-browser MCP server?
    mcp-server-rag-web-browser is a Model Context Protocol (MCP) server profile on explainx.ai. MCP lets AI hosts (e.g. Claude Desktop, Cursor) call tools and resources through a standard interface; this page summarizes categories, install hints, and community ratings.
    How do MCP servers relate to agent skills?
    Skills are reusable instruction packages (often SKILL.md); MCP servers expose live capabilities. Teams frequently combine both—skills for workflows, MCP for APIs and data. See explainx.ai/skills and explainx.ai/mcp-servers for parallel directories.
    How are reviews shown for mcp-server-rag-web-browser?
    This profile displays 29 aggregated ratings (sample rows for discoverability plus signed-in user reviews). Average score is about 4.5 out of 5—verify behavior in your own environment before production use.

    Use Cases

    Web Research & Information Gathering

    Fetch and extract information from websites automatically

    Example

    Research competitor pricing, scrape product reviews, monitor news mentions

    Automate 5-10 hours/week of manual web research

    Content Monitoring & Alerts

    Track website changes, new content, price updates

    Example

    Monitor competitor blog for new posts, track stock availability, watch for pricing changes

    Stay informed without manual checking, never miss important updates

    Data Extraction & Aggregation

    Extract structured data from multiple websites

    Example

    Compile product listings from 10 e-commerce sites, aggregate job postings, collect real estate data

    Build datasets 100x faster than manual copying

    API-less Integration

    Interact with services that don't offer APIs

    Example

    Check form submissions, validate website functionality, test user flows

    Automate interactions with any website, even without API

    Implementation Guide

    Prerequisites

    • Claude Desktop or Cursor with MCP support
    • Understanding of web scraping ethics and robots.txt
    • Rate limiting awareness to avoid overwhelming target sites
    • Knowledge of legal restrictions on data collection

    Time Estimate

    20-40 minutes including configuration and testing

    Installation Steps

    1. 1.Install web automation MCP server via npm or pip
    2. 2.Configure allowed domains and rate limits in MCP config
    3. 3.Test with simple fetch: 'Get content from example.com'
    4. 4.Progress to extraction: 'Extract all product prices from this page'
    5. 5.Set up monitoring: 'Check this URL daily for changes'
    6. 6.Parse structured data: 'Create CSV from this table'
    7. 7.Respect robots.txt and rate limits always

    Troubleshooting

    • 403 Forbidden: Website blocks bots—respect their wishes, use official API instead
    • Rate limit errors: Slow down requests, add delays between fetches
    • Stale data: Target site changed HTML structure—update selectors
    • Timeout errors: Site is slow or blocking—increase timeout, try different user agent
    • JavaScript-rendered content: Use headless browser MCP servers for dynamic sites

    Best Practices

    ✓ Do

    • +Check robots.txt and respect crawl rules
    • +Rate limit requests: 1-2 requests/second maximum
    • +Use official APIs when available instead of scraping
    • +Identify your bot with descriptive user agent
    • +Cache results to minimize repeated requests
    • +Handle errors gracefully with retries and fallbacks
    • +Validate extracted data for accuracy

    ✗ Don't

    • Don't scrape sites that explicitly forbid it (robots.txt, ToS)
    • Don't overwhelm servers with rapid requests—use rate limiting
    • Don't scrape personal data without consent and legal basis
    • Don't ignore copyright on extracted content
    • Don't assume HTML structure is stable—handle changes
    • Don't use scraped data for commercial purposes without permission

    💡 Pro Tips

    • Use CSS selectors or XPath for robust data extraction
    • Set up monitoring alerts for extraction failures (structure changed)
    • Implement exponential backoff for retries on failures
    • Store raw HTML for reprocessing if extraction logic changes
    • Combine with data analysis tools for insights from extracted data
    • Consider using official APIs or RSS feeds as more stable alternatives

    Technical Details

    Architecture

    MCP server handles HTTP requests, HTML parsing, JavaScript rendering (if headless browser), and returns structured data to Claude.

    Protocols

    • HTTP/HTTPS
    • WebSocket (for real-time sites)
    • Puppeteer/Playwright (for JavaScript sites)

    Compatibility

    • Static HTML sites
    • JavaScript-rendered SPAs (with headless browser)
    • REST APIs
    • GraphQL endpoints

    When to Use This

    ✓ Use When

    Use for research automation, content monitoring, data aggregation from multiple sources, and when official APIs don't exist. Best for read-only information gathering.

    ✗ Avoid When

    Avoid for sites with APIs (use API instead), sites that explicitly forbid scraping, when data is copyrighted, or for login-required content without proper authorization.

    Integration

    • Scheduled monitoring with change detection
    • Multi-source data aggregation pipelines
    • Fallback to web scraping when API rate limits hit
    • Headless browser for JavaScript-heavy sites

    Discussion

    Product Hunt–style comments (not star reviews)
    • No comments yet — start the thread.

    List & Promote Your MCP Server

    Share your MCP server with the developer community

    GET_STARTED →
    MCP server reviews

    Ratings

    4.529 reviews
    • Kabir White· Dec 28, 2024

      mcp-server-rag-web-browser is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.

    • Dhruvi Jain· Dec 20, 2024

      I recommend mcp-server-rag-web-browser for teams standardizing on MCP; the explainx.ai page compares cleanly with sibling servers.

    • Noah Abbas· Dec 20, 2024

      Strong directory entry: mcp-server-rag-web-browser surfaces stars and publisher context so we could sanity-check maintenance before adopting.

    • Rahul Santra· Nov 19, 2024

      Strong directory entry: mcp-server-rag-web-browser surfaces stars and publisher context so we could sanity-check maintenance before adopting.

    • Diego Zhang· Nov 19, 2024

      We evaluated mcp-server-rag-web-browser against two servers with overlapping tools; this profile had the clearer scope statement.

    • Oshnikdeep· Nov 11, 2024

      According to our notes, mcp-server-rag-web-browser benefits from clear Model Context Protocol framing — fewer ambiguous “AI plugin” claims.

    • Pratham Ware· Oct 10, 2024

      Useful MCP listing: mcp-server-rag-web-browser is the kind of server we cite when onboarding engineers to host + tool permissions.

    • Kabir Perez· Oct 10, 2024

      I recommend mcp-server-rag-web-browser for teams standardizing on MCP; the explainx.ai page compares cleanly with sibling servers.

    • Ganesh Mohane· Oct 2, 2024

      mcp-server-rag-web-browser is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.

    • Aarav Haddad· Sep 13, 2024

      We wired mcp-server-rag-web-browser into a staging workspace; the listing’s GitHub and npm pointers saved time versus hunting across READMEs.

    showing 1-10 of 29

    1 / 3