explainx.ainewsletter3.4k
Backend

apify-scrapers▌

casper-studios/casper-marketplace · updated Apr 8, 2026

MDX-style export adds YAML metadata + attribution linking explainx.ai and this canonical listing URL.

$npx skills add https://github.com/casper-studios/casper-marketplace --skill apify-scrapers
0 commentsdiscussion
summary

Scrape content from major social platforms using Apify actors. Each platform has optimized settings for cost and quality.

skill.md

Apify Scrapers

Overview

Scrape content from major social platforms using Apify actors. Each platform has optimized settings for cost and quality.

Quick Decision Tree

What do you want to scrape?
│
├── Social Media Posts
│   ├── Twitter/X → references/twitter.md
│   │   └── Script: scripts/scrape_twitter_ai_trends.py
│   │
│   ├── Reddit → references/reddit.md
│   │   └── Script: scripts/scrape_reddit_ai_tech.py
│   │
│   ├── LinkedIn → references/linkedin.md
│   │   └── Script: scripts/scrape_linkedin_posts.py
│   │
│   ├── Instagram → references/instagram.md
│   │   └── Script: scripts/scrape_instagram.py
│   │   └── Modes: profile, posts, hashtag, reels, comments
│   │
│   ├── Facebook → references/facebook.md
│   │   └── Script: scripts/scrape_facebook.py
│   │   └── Modes: page, posts, reviews, groups, marketplace
│   │
│   ├── TikTok → references/multi-platform.md
│   │   └── Script: scripts/scrape_multi_platform.py
│   │
│   └── YouTube → references/multi-platform.md
│       └── Script: scripts/scrape_multi_platform.py
│
├── Business/Places
│   ├── Google Maps businesses → references/google-maps.md
│   │   └── Script: scripts/scrape_google_maps.py
│   │   └── Modes: search, place, reviews
│   │
│   └── Contact info from websites → references/contact-enrichment.md
│       └── Script: scripts/scrape_contact_info.py
│       └── Extract: emails, phone numbers, social profiles
│
├── Auto-detect URL type → references/url-detect.md
│   └── Script: scripts/scrape_content_by_url.py
│
├── Trend Analysis (NEW)
│   └── Enriched trend analysis → workflows/trend-analysis.md
│       └── Script: scripts/analyze_trends.py
│       └── Features: velocity scoring, lifecycle staging, opportunity scoring
│
└── Workflows (multi-step)
    ├── Lead generation → workflows/lead-generation.md
    ├── Influencer discovery → workflows/influencer-discovery.md
    ├── Competitor analysis → workflows/competitor-intel.md
    ├── Trend analysis → workflows/trend-analysis.md
    └── Competitor Ads Intelligence (NEW) → workflows/competitor-ads.md
        └── Script: scripts/scrape_competitor_ads.py
        └── Platforms: Facebook Ads Library, Google Ads Transparency
        └── Features: Spend estimates, creative analysis, benchmarking

Environment Setup

# Required in .env
APIFY_TOKEN=apify_api_xxxxx

Get your API key: https://console.apify.com/account/integrations

Common Usage Patterns

Scrape Twitter Trends

python scripts/scrape_twitter_ai_trends.py --query "AI agents" --max-tweets 50

Scrape Reddit Discussions

python scripts/scrape_reddit_ai_tech.py --subreddits "MachineLearning,LocalLLaMA" --max-posts 100

Scrape LinkedIn Author

python scripts/scrape_linkedin_posts.py author "https://linkedin.com/in/username" --max-posts 30

Auto-detect and Scrape URL

python scripts/scrape_content_by_url.py "https://x.com/user/status/123456"

Scrape Instagram Profile

python scripts/scrape_instagram.py profile "https://instagram.com/username" --max-posts 20

Scrape Instagram Hashtag

python scripts/scrape_instagram.py hashtag "#artificialintelligence" --max-posts 50

Scrape Instagram Reels

python scripts/scrape_instagram.py reels "https://instagram.com/username" --max-reels 30

Scrape Facebook Page

python scripts/scrape_facebook.py page "https://facebook.com/pagename" --max-posts 50

Scrape Facebook Reviews

python scripts/scrape_facebook.py reviews "https://facebook.com/pagename" --max-reviews 100

Scrape Facebook Marketplace

python scripts/scrape_facebook.py marketplace "laptops in san francisco" --max-items 30

Scrape Google Maps Businesses

python scripts/scrape_google_maps.py search "AI consulting firms in New York" --max-results 50

Scrape Google Maps Reviews

python scripts/scrape_google_maps.py reviews "ChIJN1t_tDeuEmsRUsoyG83frY4" --max-reviews 100

Extract Contact Info from Websites

python scripts/scrape_contact_info.py "https://example.com" --depth 2

Bulk Contact Enrichment

python scripts/scrape_contact_info.py --urls-file companies.txt --output contacts.json

Scrape Competitor Ads (Single Competitor)

python scripts/scrape_competitor_ads.py "Nike" --platforms facebook google --country US --days 30

Compare Multiple Competitors' Ads

python scripts/scrape_competitor_ads.py "Nike" "Adidas" "Puma" --compare --output comparison.json

Discover Advertisers by Keyword

python scripts/scrape_competitor_ads.py --search "running shoes" --country US --max-ads 200

Filter Competitor Ads by Media Type

python scripts/scrape_competitor_ads.py "Netflix" "Disney+" --platforms facebook --media-types video --days 7

Analyze Trends (NEW)

# Analyze specific topic with enrichments
python scripts/analyze_trends.py "artificial intelligence" --sources google instagram tiktok --days 90

# Discover trending topics in category
python scripts/analyze_trends.py --category technology --discover --top 50

# Compare multiple trends
python scripts/analyze_trends.py "AI" "blockchain" "metaverse" --compare

# Export HTML trend report
python scripts/analyze_trends.py "sustainable fashion" --format html --output trend_report.html

Cost Estimates

Platform Actor Cost per Item
Twitter kaitoeasyapi/twitter-x-data-tweet-scraper ~$0.00025
Reddit trudax/reddit-scraper ~$0.001-0.005
LinkedIn harvestapi/linkedin-post-search ~$0.01-0.05
YouTube streamers/youtube-scraper ~$0.01-0.05
TikTok clockworks/tiktok-scraper ~$0.005
Instagram (profile) apify/instagram-profile-scraper ~$0.005
Instagram (posts) apify/instagram-post-scraper ~$0.002-0.005
Instagram (hashtag) apify/instagram-hashtag-scraper ~$0.002-0.005
Instagram (reels) apify/instagram-reel-scraper ~$0.005-0.01
Instagram (comments) apify/instagram-comment-scraper ~$0.001-0.003
Facebook (page) apify/facebook-pages-scraper ~$0.005-0.01
Facebook (posts) apify/facebook-posts-scraper ~$0.003-0.005
Facebook (reviews) apify/facebook-reviews-scraper ~$0.002-0.005
Facebook (groups) apify/facebook-groups-scraper ~$0.005-0.01
Facebook (marketplace) apify/facebook-marketplace-scraper ~$0.005-0.01
Google Maps (search) compass/crawler-google-places ~$0.01-0.02
Google Maps (place) compass/google-maps-business-scraper ~$0.01
Google Maps (reviews) compass/google-maps-reviews-scraper ~$0.003-0.005
Contact Enrichment lukaskrivka/contact-info-scraper ~$0.01-0.03
Google Trends apify/google-trends-scraper ~$0.01
Trend Analysis (multi) Multiple actors ~$0.50-1.50/run
Facebook Ads Library apify/facebook-ads-scraper ~$0.75/1K ads
Facebook Ads (alt) curious_coder/facebook-ads-library-scraper ~$0.50/1K ads
Google Ads Transparency lexis-solutions/google-ads-scraper ~$1.00/1K ads
Google Ads (alt) xtech/google-ad-transparency-scraper ~$0.80/1K ads

Output Location

All scraped data saves to .tmp/ with timestamped filenames:

  • .tmp/twitter_ai_trends_YYYYMMDD.json
  • .tmp/reddit_ai_tech_YYYYMMDD.json
  • .tmp/linkedin_posts_YYYYMMDD_HHMMSS.json

Security Notes

Credential Handling

  • Store APIFY_TOKEN in .env file (never commit to git)
  • Rotate API tokens periodically via Apify Console
  • Never log or print API tokens in script output
  • Use environment variables, not hardcoded values

Data Privacy

  • Scraped data contains only publicly available content
  • Social media posts may include PII (names, handles, profile info)
  • Data is stored locally in .tmp/ directory
  • No data is retained by Apify after actor run completes
  • Consider data minimization - only scrape what you need

Access Scopes

  • Apify tokens have full account access (no granular scopes)
  • Use separate Apify accounts for different projects if needed
  • Monitor usage via Apify Console dashboard

Compliance Considerations

  • Terms of Service: Respect each platform's ToS (Twitter, Reddit, LinkedIn)
  • Rate Limiting: Actors have built-in rate limiting to avoid bans
  • Robots.txt: Some actors may bypass robots.txt - use responsibly
  • GDPR: Scraped PII may be subject to GDPR if EU residents
  • Ethical Use: Only scrape public data; never bypass authentication
  • Proxy Ethics: Residential proxies should be used ethically

Troubleshooting

Common Issues

Issue: Actor run failed

Symptoms: Script terminates with "Actor run failed" or timeout error Cause: Invalid actor ID, insufficient proxy credits, or actor configuration issue Solution:

  • Verify the actor ID is correct in the script
  • Check Apify Console for actor run logs
  • Ensure proxy settings match actor requirements
  • Try running with default proxy settings first

Issue: Empty results returned

Symptoms: Script completes but returns 0 items Cause: Content blocked by platform, invalid query, or proxy being detected Solution:

  • Try a different proxy type (residential vs datacenter)
  • Simplify the search query
  • Reduce the number of results requested
  • Check if the platform is blocking scraping attempts

Issue: Rate limited by platform

Symptoms: Script fails with 429 errors or "rate limited" messages Cause: Too many requests in a short time period Solution:

  • Add delays between requests (actor settings)
  • Reduce concurrent requests
  • Use proxy rotation
  • Wait and retry after a cooldown period

Issue: Invalid API token

Symptoms: Authentication error or "invalid token" message Cause: Token expired, revoked, or incorrectly set Solution:

  • Regenerate API token in Apify Console
  • Verify token is correctly set in .env file
  • Check for leading/trailing whitespace in token
  • Ensure APIFY_TOKEN environment variable is loaded

Issue: Proxy connection errors

Symptoms: Connection timeout or proxy errors Cause: Proxy pool exhausted or geo-restriction issues Solution:

  • Switch proxy type (basic, residential, or datacenter)
  • Verify proxy credit balance in Apify Console
  • Try a different proxy country/region
  • Disable proxy to test if that's the root cause

Resources

Platform References

  • references/twitter.md - Twitter/X scraping details
  • references/reddit.md - Reddit scraping with subreddit targeting
  • references/linkedin.md - LinkedIn post scraping (author or search mode)
  • references/instagram.md - Instagram profile, posts, hashtag, reels, and comments scraping
  • references/facebook.md - Facebook page, posts, reviews, groups, and marketplace scraping
  • references/multi-platform.md - TikTok and YouTube scraping
  • references/url-detect.md - Auto-detect URL type and scrape

Business/Places References

  • references/google-maps.md - Google Maps business search, place details, and reviews
  • references/contact-enrichment.md - Extract emails, phone numbers, and social profiles from websites

Workflow References

  • workflows/lead-generation.md - Multi-step lead generation workflow
  • workflows/influencer-discovery.md - Find and analyze influencers across platforms
  • workflows/competitor-intel.md - Competitive intelligence gathering workflow
  • workflows/trend-analysis.md - Enriched multi-platform trend analysis with scoring

Integration Patterns

Scrape and Enrich

Skills: apify-scrapers → parallel-research Use case: Scrape social media posts, then enrich with deep research Flow:

  1. Scrape Twitter/Reddit for mentions of a topic
  2. Extract company names or URLs from posts
  3. Use parallel-research to get detailed info on each company

Scrape and Summarize

Skills: apify-scrapers → content-generation Use case: Create newsletter content from social media trends Flow:

  1. Scrape trending AI posts from Twitter
  2. Pass scraped data to content-generation summarize
  3. Generate a formatted newsletter section

Scrape and Archive

Skills: apify-scrapers → google-workspace Use case: Save scraped data to Google Drive for team access Flow:

  1. Scrape LinkedIn posts from target accounts
  2. Format data as CSV or JSON
  3. Upload to Google Drive client folder via google-workspace

Trend Analysis + Content Strategy

Skills: apify-scrapers (trend-analysis) → content-generation Use case: Identify trending topics and create content strategy Flow:

  1. Run trend analysis: python scripts/analyze_trends.py "AI productivity" --sources all
  2. Review lifecycle stage and opportunity score
  3. Use content-generation to create content for high-opportunity trends
  4. Focus on emerging trends with high velocity scores

Competitive Trend Monitoring

Skills: apify-scrapers (trend-analysis) → parallel-research Use case: Monitor competitor visibility in trending topics Flow:

  1. Analyze industry trends: python scripts/analyze_trends.py --category "your-industry" --discover
  2. Compare your brand vs competitors in those trends
  3. Use parallel-research for deep dive on gaps
  4. Generate competitive intelligence report
how to use apify-scrapers

How to use apify-scrapers on Cursor

AI-first code editor with Composer

1

Prerequisites

Before installing skills in Cursor, ensure your development environment meets these requirements:

  • ›Cursor installed and configured on your development machine
  • ›Node.js version 16.0+ with npm package manager (verify with node --version)
  • ›Active project directory or workspace where you want to add apify-scrapers
2

Execute installation command

Execute the skills CLI command in your project's root directory to begin installation:

$npx skills add https://github.com/casper-studios/casper-marketplace --skill apify-scrapers

The skills CLI fetches apify-scrapers from GitHub repository casper-studios/casper-marketplace and configures it for Cursor.

3

Select Cursor when prompted

The CLI will show a list of available agents. Use arrow keys to navigate and space to select Cursor:

◆ Which agents do you want to install to?
│
│ ── Universal (.agents/skills) ── always included ────
│ • Amp
│ • Antigravity
│ • Cline
│ • Codex
│ ●Cursor(selected)
│ • Cursor
│ • Windsurf
4

Verify installation

Confirm successful installation by checking the skill directory location:

.cursor/skills/apify-scrapers

Reload or restart Cursor to activate apify-scrapers. Access the skill through slash commands (e.g., /apify-scrapers) or your agent's skill management interface.

⚠

Security & Verification Notice

We perform automated surface-level scans (Gen AI Scanner, Socket, Snyk) during installation. These checks detect common vulnerabilities but do not guarantee complete security. Always review skill source code and verify the publisher's reputation before production use.

Skills execute code in your development environment. Always verify the publisher's identity, review recent commits, and test in isolated environments before production deployment.

Additional Resources

›View source code on GitHub›Skills CLI documentation›Learn more about Cursor›What are agent skills?

List & Monetize Your Skill

Submit your Claude Code skill and start earning

GET_STARTED →

Use Cases▌

Task Automation & Efficiency

Automate repetitive workflows and reduce manual effort

Example

Generate reports, summarize documents, draft communications

✓

Save 3-5 hours per week on routine tasks

Knowledge Enhancement

Learn new skills, understand complex topics, get expert guidance

Example

Explain concepts, provide examples, suggest learning resources

✓

Accelerate learning and skill development by 2x

Quality Improvement

Enhance output quality through reviews, suggestions, and refinements

Example

Review drafts, suggest improvements, catch errors

✓

Improve work quality by 30-40% with less effort

Implementation Guide▌

Prerequisites

  • ›Claude Desktop or compatible AI client with skill support
  • ›Clear understanding of task or problem to solve
  • ›Willingness to iterate and refine outputs

Time Estimate

15-45 minutes depending on use case complexity

Installation Steps

  1. 1.Install skill using provided installation command
  2. 2.Test with simple use case relevant to your work
  3. 3.Evaluate output quality and relevance
  4. 4.Iterate on prompts to improve results
  5. 5.Integrate into regular workflow if valuable

Common Pitfalls

  • ⚠Expecting perfect results without iteration
  • ⚠Not providing enough context in prompts
  • ⚠Using skill for tasks outside its intended scope
  • ⚠Accepting outputs without review and validation

Best Practices▌

✓ Do

  • +Start with clear, specific prompts
  • +Provide relevant context and constraints
  • +Review and refine all outputs before using
  • +Iterate to improve output quality
  • +Document successful prompt patterns

✗ Don't

  • −Don't use without understanding skill limitations
  • −Don't skip validation of outputs
  • −Don't share sensitive information in prompts
  • −Don't expect skill to replace human judgment

💡 Pro Tips

  • ★Be specific about desired format and style
  • ★Ask for multiple options to choose from
  • ★Request explanations to understand reasoning
  • ★Combine AI efficiency with human expertise

When to Use This▌

✓ Use When

Use when skill capabilities match your task, clear ROI on time saved, and you can validate outputs. Best for repetitive tasks, learning, and quality improvement.

✗ Avoid When

Avoid when task requires deep expertise you can't validate, involves sensitive decisions, or when learning process is more valuable than speed of completion.

Learning Path▌

  1. 1Familiarize yourself with skill capabilities and limitations
  2. 2Start with low-risk, non-critical tasks
  3. 3Progress to more complex and valuable use cases
  4. 4Build expertise through regular use and experimentation

Discussion

Product Hunt–style comments (not star reviews)
  • No comments yet — start the thread.
general reviews

Ratings

4.8★★★★★33 reviews
  • ★★★★★Neel Verma· Dec 16, 2024

    I recommend apify-scrapers for anyone iterating fast on agent tooling; clear intent and a small, reviewable surface area.

  • ★★★★★Chaitanya Patil· Dec 12, 2024

    Registry listing for apify-scrapers matched our evaluation — installs cleanly and behaves as described in the markdown.

  • ★★★★★Ren Zhang· Dec 4, 2024

    Solid pick for teams standardizing on skills: apify-scrapers is focused, and the summary matches what you get after install.

  • ★★★★★Ren Huang· Nov 23, 2024

    apify-scrapers has been reliable in day-to-day use. Documentation quality is above average for community skills.

  • ★★★★★Luis Brown· Nov 15, 2024

    apify-scrapers is among the better-maintained entries we tried; worth keeping pinned for repeat workflows.

  • ★★★★★Camila Nasser· Nov 7, 2024

    Keeps context tight: apify-scrapers is the kind of skill you can hand to a new teammate without a long onboarding doc.

  • ★★★★★Piyush G· Nov 3, 2024

    apify-scrapers reduced setup friction for our internal harness; good balance of opinion and flexibility.

  • ★★★★★Camila Wang· Oct 26, 2024

    Registry listing for apify-scrapers matched our evaluation — installs cleanly and behaves as described in the markdown.

  • ★★★★★Shikha Mishra· Oct 22, 2024

    I recommend apify-scrapers for anyone iterating fast on agent tooling; clear intent and a small, reviewable surface area.

  • ★★★★★Ava Thomas· Oct 14, 2024

    Useful defaults in apify-scrapers — fewer surprises than typical one-off scripts, and it plays nicely with `npx skills` flows.

showing 1-10 of 33

1 / 4