Firecrawl Web Scraper Skill

Status: Production Ready Last Updated: 2026-01-20 Official Docs: https://docs.firecrawl.dev API Version: v2 SDK Versions: firecrawl-py 4.13.0+, @mendable/firecrawl-js 4.11.1+

What is Firecrawl?

Firecrawl is a Web Data API for AI that turns websites into LLM-ready markdown or structured data. It handles:

• JavaScript rendering - Executes client-side JavaScript to capture dynamic content
• Anti-bot bypass - Gets past CAPTCHA and bot detection systems
• Format conversion - Outputs as markdown, HTML, JSON, screenshots, summaries
• Document parsing - Processes PDFs, DOCX files, and images
• Autonomous agents - AI-powered web data gathering without URLs
• Change tracking - Monitor content changes over time
• Branding extraction - Extract color schemes, typography, logos

API Endpoints Overview

1. Scrape Endpoint (/v2/scrape)

Scrapes a single webpage and returns clean, structured content.

Basic Usage

from firecrawl import Firecrawl
import os

app = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY"))

# Basic scrape
doc = app.scrape(
    url="https://example.com/article",
    formats=["markdown", "html"],
    only_main_content=True
)

print(doc.markdown)
print(doc.metadata)

import FirecrawlApp from '@mendable/firecrawl-js';

const app = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY });

const result = await app.scrapeUrl('https://example.com/article', {
  formats: ['markdown', 'html'],
  onlyMainContent: true
});

console.log(result.markdown);

Output Formats

Advanced Options

doc = app.scrape(
    url="https://example.com",
    formats=["markdown", "screenshot"],
    only_main_content=True,
    remove_base64_images=True,
    wait_for=5000,  # Wait 5s for JS
    timeout=30000,
    # Location & language
    location={"country": "AU", "languages": ["en-AU"]},
    # Cache control
    max_age=0,  # Fresh content (no cache)
    store_in_cache=True,
    # Stealth mode for complex sites
    stealth=True,
    # Custom headers
    headers={"User-Agent": "Custom Bot 1.0"}
)

Browser Actions

Perform interactions before scraping:

doc = app.scrape(
    url="https://example.com",
    actions=[
        {"type": "click", "selector": "button.load-more"},
        {"type": "wait", "milliseconds": 2000},
        {"type": "scroll", "direction": "down"},
        {"type": "write", "selector": "input#search", "text": "query"},
        {"type": "press", "key": "Enter"},
        {"type": "screenshot"}  # Capture state mid-action
    ]
)

JSON Mode (Structured Extraction)

# With schema
doc = app.scrape(
    url="https://example.com/product",
    formats=["json"],
    json_options={
        "schema": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "price": {"type": "number"},
                "in_stock": {"type": "boolean"}
            }
        }
    }
)

# Without schema (prompt-only)
doc = app.scrape(
    url="https://example.com/product",
    formats=["json"],
    json_options={
        "prompt": "Extract the product name, price, and availability"
    }
)

Branding Extraction

Extract design system and brand identity:

doc = app.scrape(
    url="https://example.com",
    formats=["branding"]
)

# Returns:
# - Color schemes and palettes
# - Typography (fonts, sizes, weights)
# - Spacing and layout metrics
# - UI component styles
# - Logo and imagery URLs
# - Brand personality traits

2. Crawl Endpoint (/v2/crawl)

Crawls all accessible pages from a starting URL.

result = app.crawl(
    url="https://docs.example.com",
    limit=100,
    max_depth=3,
    allowed_domains=["docs.example.com"],
    exclude_paths=["/api/*", "/admin/*"],
    scrape_options={
        "formats": ["markdown"],
        "only_main_content": True
    }
)

for page in result.data:
    print(f"Scraped: {page.metadata.source_url}")
    print(f"Content: {page.markdown[:200]}...")

Async Crawl with Webhooks

# Start crawl (returns immediately)
job = app.start_crawl(
    url="https://docs.example.com",
    limit