search-web

4get

yshalsager

by yshalsager

4get is a privacy-focused private search engine that aggregates web, image, and news results while protecting your data

Integrates with the 4get meta search engine to provide privacy-focused web, image, and news searches through a search aggregator that maintains user privacy while accessing results from multiple sources with built-in rate limiting resilience and caching optimization.

github stars

5

0 commentsdiscussion

Both formats append explainx.ai attribution and the canonical URL for this MCP server listing.

Privacy-focused with no user trackingBuilt-in rate limiting and cachingAggregates multiple search engines

best for

  • / Privacy-conscious users avoiding tracking
  • / Research requiring diverse search sources
  • / Content creators needing image searches

capabilities

  • / Search the web across multiple search engines
  • / Find images from various sources
  • / Search news articles
  • / Access cached and optimized results

what it does

Provides privacy-focused web searches through the 4get meta search engine, which aggregates results from multiple sources without tracking users.

about

4get is a community-built MCP server published by yshalsager that provides AI assistants with tools and capabilities via the Model Context Protocol. 4get is a privacy-focused private search engine that aggregates web, image, and news results while protecting your data It is categorized under search web.

how to install

You can install 4get in your AI client of choice. Use the install panel on this page to get one-click setup for Cursor, Claude Desktop, VS Code, and other MCP-compatible clients. This server runs locally on your machine via the stdio transport.

license

GPL-3.0

4get is released under the GPL-3.0 license.

readme

4get MCP Server

A MCP server that provides seamless access to the 4get Meta Search engine API for LLM clients via FastMCP.

Codacy Badge Codacy Badge PyPI version PyPI Downloads

GitHub release GitHub Downloads

made-with-python Open Source Love

PayPal LiberaPay

✨ Features

  • 🔍 Multi Search Functions: Web, image, and news search with comprehensive result formatting
  • ⚡ Smart Caching: TTL-based response caching with configurable size limits
  • 🔄 Retry Logic: Exponential backoff for rate-limited and network errors
  • 🏗️ Production Ready: Connection pooling, comprehensive error handling, and validation
  • 📊 Rich Responses: Featured answers, related searches, pagination support, and more
  • 🧪 Well Tested: Extensive test suite including integration tests with real API, unit tests, and more
  • ⚙️ Highly Configurable: 11+ environment variables for fine-tuning
  • 🎯 Engine Shorthands: Pick a 4get scraper via the engine parameter without memorizing query strings

📋 Requirements

  • Python 3.13+
  • uv for dependency management

Quick Start

# Install dependencies
uv sync

# Run the server
uv run -m mcp_4get

# Or use mise
mise run

⚙️ Configuration

The server is highly configurable via environment variables. All settings have sensible defaults for the public https://4get.ca instance.

Core Settings

VariableDescriptionDefault
FOURGET_BASE_URLBase URL for the 4get instancehttps://4get.ca
FOURGET_PASSOptional pass token for rate-limited instancesunset
FOURGET_USER_AGENTOverride User-Agent headermcp-4get/<version>
FOURGET_TIMEOUTRequest timeout in seconds20.0

Caching & Performance

VariableDescriptionDefault
FOURGET_CACHE_TTLCache lifetime in seconds600.0
FOURGET_CACHE_MAXSIZEMaximum cached responses128
FOURGET_CONNECTION_POOL_MAXSIZEMax concurrent connections10
FOURGET_CONNECTION_POOL_MAX_KEEPALIVEMax persistent connections5

Retry & Resilience

VariableDescriptionDefault
FOURGET_MAX_RETRIESMaximum retry attempts3
FOURGET_RETRY_BASE_DELAYBase retry delay in seconds1.0
FOURGET_RETRY_MAX_DELAYMaximum retry delay in seconds60.0

🚀 Running the Server

Local Development

uv run -m mcp_4get

Production Deployment

# With custom configuration
export FOURGET_BASE_URL="https://my-4get-instance.com"
export FOURGET_PASS="my-secret-token"
export FOURGET_CACHE_TTL="300"
export FOURGET_MAX_RETRIES="5"

uv run -m mcp_4get

MCP Server Integration

You can integrate the 4get MCP server with popular IDEs and AI assistants. Here are configuration examples:

Cursor IDE

Add this to your Cursor MCP configuration (~/.cursor/mcp.json):

{
  "mcpServers": {
    "4get": {
      "command": "uvx",
      "args": [
        "mcp_4get@latest"
      ],
      "env": {
        "FOURGET_BASE_URL": "https://4get.ca"
      }
    }
  }
}

OpenAI Codex

Add this to your Codex MCP configuration (~/.codex/config.toml):

[mcp_servers.4get]
command = "uvx"
args = ["mcp_4get@latest"]
env = { FOURGET_BASE_URL = "https://4get.ca" }

Note: Replace /path/to/your/mcp-4get with the actual path to your project directory.

🔧 MCP Tools

The server exposes three powerful search tools with comprehensive response formatting:

fourget_web_search

fourget_web_search(
    query: str,
    page_token: str = None,        # Use 'npt' from previous response
    extended_search: bool = False, # Enable extended search mode
    engine: str = None,             # Pick a scraper from the supported engine list
    extra_params: dict = None      # Language, region, etc.
)

Response includes: web[], answer[], spelling, related[], npt

fourget_image_search

fourget_image_search(
    query: str,
    page_token: str = None,   # Use 'npt' from previous response
    engine: str = None,       # Pick a scraper from the supported engine list
    extra_params: dict = None # Size, color, type filters
)

Response includes: image[], npt

fourget_news_search

fourget_news_search(
    query: str,
    page_token: str = None,   # Use 'npt' from previous response
    engine: str = None,       # Pick a scraper from the supported engine list
    extra_params: dict = None # Date range, source filters
)

Response includes: news[], npt

Engine shorthands

All MCP tools accept an optional engine argument that maps directly to the 4get scraper query parameter. This shorthand overrides any scraper value you may include in extra_params.

ValueEngine
ddgDuckDuckGo
braveBrave
mullvad_braveMullvad (Brave)
yandexYandex
googleGoogle
google_cseGoogle CSE
mullvad_googleMullvad (Google)
startpageStartpage
qwantQwant
ghosteryGhostery
yepYep
grepprGreppr
crowdviewCrowdview
mwmblMwmbl
mojeekMojeek
baiduBaidu
coccocCoc Coc
solofieldSolofield
marginaliaMarginalia
wibywiby
curlieCurlie

If you need to pass additional 4get query parameters (such as country or language), continue to supply them through extra_params.

📄 Pagination

All tools support pagination via the npt (next page token):

# Get first page
result = await client.web_search("python programming")

# Get next page if available
if result.get('npt'):
    next_page = await client.web_search("ignored", page_token=result['npt'])

🐍 Using the Async Client Directly

You can reuse the bundled async client outside MCP for direct API access:

import asyncio
from mcp_4get.client import FourGetClient
from mcp_4get.config import Config

async def main() -> None:
    client = FourGetClient(Config.from_env())
    data = await client.web_search(
        "model context protocol",
        options={"scraper": "mullvad_brave"},
    )
    for result in data.get("web", []):
        print(result["title"], "->", result["url"])

asyncio.run(main())

This allows you to integrate 4get search capabilities directly into your Python applications without going through the MCP protocol.

🛡️ Error Handling & Resilience

Automatic Retry Logic

  • Rate Limiting (429): Exponential backoff with jitter
  • Network Errors: Connection failures and timeouts
  • Non-retryable: HTTP 404/500 errors fail immediately

Error Types

  • FourGetAuthError: Rate limited or invalid authentication
  • FourGetAPIError: API returned non-success status
  • FourGetTransportError: Network or HTTP protocol errors
  • FourGetError: Generic client errors

Configuration Validation

All settings are validated on startup with clear error messages for misconfigurations.

📊 Response Format

Based on the real 4get API, responses include rich metadata:

{
  "status": "ok",
  "web": [
    {
      "title": "Example Result",
      "description": "Result description...",
      "url": "https://example.com",
      "date": 1640995200,
      "type": "web"
    }
  ],
  "answer": [
    {
      "title": "Featured Answer",
      "description": [{"type": "text", "value": "Answer content..."}],
      "url": "https://source.com",
      "table": {"Key": "Value"}
    }
  ],
  "spelling": {
    "type": "no_correction",
    "correction": null
  },
  "related": ["related search", "terms"],
  "npt": "pagination_token_here"
}

Development

This project uses several tools to streamline the development process:

mise

mise is used for managing project-level dependencies and environment variables. mise helps ensure consistent development environments across different machines.

To get started with mise:

  1. Install mise by following the instructions on the official website.
  2. Run mise install in the project root to set up the development environment.

**Environment Variab


FAQ

What is the 4get MCP server?
4get is a Model Context Protocol (MCP) server profile on explainx.ai. MCP lets AI hosts (e.g. Claude Desktop, Cursor) call tools and resources through a standard interface; this page summarizes categories, install hints, and community ratings.
How do MCP servers relate to agent skills?
Skills are reusable instruction packages (often SKILL.md); MCP servers expose live capabilities. Teams frequently combine both—skills for workflows, MCP for APIs and data. See explainx.ai/skills and explainx.ai/mcp-servers for parallel directories.
How are reviews shown for 4get?
This profile displays 64 aggregated ratings (sample rows for discoverability plus signed-in user reviews). Average score is about 4.7 out of 5—verify behavior in your own environment before production use.

Use Cases

Web Research & Information Gathering

Fetch and extract information from websites automatically

Example

Research competitor pricing, scrape product reviews, monitor news mentions

Automate 5-10 hours/week of manual web research

Content Monitoring & Alerts

Track website changes, new content, price updates

Example

Monitor competitor blog for new posts, track stock availability, watch for pricing changes

Stay informed without manual checking, never miss important updates

Data Extraction & Aggregation

Extract structured data from multiple websites

Example

Compile product listings from 10 e-commerce sites, aggregate job postings, collect real estate data

Build datasets 100x faster than manual copying

API-less Integration

Interact with services that don't offer APIs

Example

Check form submissions, validate website functionality, test user flows

Automate interactions with any website, even without API

Implementation Guide

Prerequisites

  • Claude Desktop or Cursor with MCP support
  • Understanding of web scraping ethics and robots.txt
  • Rate limiting awareness to avoid overwhelming target sites
  • Knowledge of legal restrictions on data collection

Time Estimate

20-40 minutes including configuration and testing

Installation Steps

  1. 1.Install web automation MCP server via npm or pip
  2. 2.Configure allowed domains and rate limits in MCP config
  3. 3.Test with simple fetch: 'Get content from example.com'
  4. 4.Progress to extraction: 'Extract all product prices from this page'
  5. 5.Set up monitoring: 'Check this URL daily for changes'
  6. 6.Parse structured data: 'Create CSV from this table'
  7. 7.Respect robots.txt and rate limits always

Troubleshooting

  • 403 Forbidden: Website blocks bots—respect their wishes, use official API instead
  • Rate limit errors: Slow down requests, add delays between fetches
  • Stale data: Target site changed HTML structure—update selectors
  • Timeout errors: Site is slow or blocking—increase timeout, try different user agent
  • JavaScript-rendered content: Use headless browser MCP servers for dynamic sites

Best Practices

✓ Do

  • +Check robots.txt and respect crawl rules
  • +Rate limit requests: 1-2 requests/second maximum
  • +Use official APIs when available instead of scraping
  • +Identify your bot with descriptive user agent
  • +Cache results to minimize repeated requests
  • +Handle errors gracefully with retries and fallbacks
  • +Validate extracted data for accuracy

✗ Don't

  • Don't scrape sites that explicitly forbid it (robots.txt, ToS)
  • Don't overwhelm servers with rapid requests—use rate limiting
  • Don't scrape personal data without consent and legal basis
  • Don't ignore copyright on extracted content
  • Don't assume HTML structure is stable—handle changes
  • Don't use scraped data for commercial purposes without permission

💡 Pro Tips

  • Use CSS selectors or XPath for robust data extraction
  • Set up monitoring alerts for extraction failures (structure changed)
  • Implement exponential backoff for retries on failures
  • Store raw HTML for reprocessing if extraction logic changes
  • Combine with data analysis tools for insights from extracted data
  • Consider using official APIs or RSS feeds as more stable alternatives

Technical Details

Architecture

MCP server handles HTTP requests, HTML parsing, JavaScript rendering (if headless browser), and returns structured data to Claude.

Protocols

  • HTTP/HTTPS
  • WebSocket (for real-time sites)
  • Puppeteer/Playwright (for JavaScript sites)

Compatibility

  • Static HTML sites
  • JavaScript-rendered SPAs (with headless browser)
  • REST APIs
  • GraphQL endpoints

When to Use This

✓ Use When

Use for research automation, content monitoring, data aggregation from multiple sources, and when official APIs don't exist. Best for read-only information gathering.

✗ Avoid When

Avoid for sites with APIs (use API instead), sites that explicitly forbid scraping, when data is copyrighted, or for login-required content without proper authorization.

Integration

  • Scheduled monitoring with change detection
  • Multi-source data aggregation pipelines
  • Fallback to web scraping when API rate limits hit
  • Headless browser for JavaScript-heavy sites

Discussion

Product Hunt–style comments (not star reviews)
  • No comments yet — start the thread.

List & Promote Your MCP Server

Share your MCP server with the developer community

GET_STARTED →
MCP server reviews

Ratings

4.764 reviews
  • Ama Huang· Dec 28, 2024

    4get is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.

  • Anaya Thomas· Dec 24, 2024

    4get has been reliable for tool-calling workflows; the MCP profile page is a good permalink for internal docs.

  • Kwame Yang· Dec 20, 2024

    We wired 4get into a staging workspace; the listing’s GitHub and npm pointers saved time versus hunting across READMEs.

  • Henry Abbas· Dec 20, 2024

    Useful MCP listing: 4get is the kind of server we cite when onboarding engineers to host + tool permissions.

  • Carlos Okafor· Dec 16, 2024

    4get is a well-scoped MCP server in the explainx.ai directory — install snippets and categories matched our Claude Code setup.

  • Shikha Mishra· Dec 12, 2024

    4get reduced integration guesswork — categories and install configs on the listing matched the upstream repo.

  • Ganesh Mohane· Dec 8, 2024

    According to our notes, 4get benefits from clear Model Context Protocol framing — fewer ambiguous “AI plugin” claims.

  • Anaya Li· Dec 4, 2024

    4get is a well-scoped MCP server in the explainx.ai directory — install snippets and categories matched our Claude Code setup.

  • Sofia Ghosh· Nov 23, 2024

    4get is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.

  • Noah Diallo· Nov 19, 2024

    4get is a well-scoped MCP server in the explainx.ai directory — install snippets and categories matched our Claude Code setup.

showing 1-10 of 64

1 / 7