web-infra-dev/midscene-skills▌

CRITICAL RULES — VIOLATIONS WILL BREAK THE WORKFLOW:

Vision-driven iOS automation using natural language commands and screenshot analysis. \n \n Operates entirely from screenshots without requiring DOM access or accessibility labels; can interact with any visible UI element regardless of technology stack \n Requires a configured vision model (Gemini, Qwen, Doubao, or similar) via environment variables for AI-powered screen understanding and action execution \n Follows a synchronous workflow: connect device, take screenshot, execute actions via nat

Vision-driven browser automation using your real Chrome browser, preserving sessions and login state. \n \n Operates entirely from screenshots without requiring DOM access; interacts with all visible elements regardless of technology stack \n Connects to desktop Chrome via Midscene Extension through Chrome DevTools Protocol, never taking over mouse or keyboard \n Supports multi-step workflows including navigation, form filling, data extraction, UI testing, and screenshot capture \n Requires visu

Vision-driven desktop automation for native apps using natural language commands and screenshots. \n \n Controls macOS, Windows, and Linux desktops entirely from visual input; no DOM or accessibility labels required \n Operates synchronously with a screenshot-analyze-act loop: connect, observe screen state, execute high-level actions via natural language prompts, then disconnect \n Requires a vision-capable AI model (Gemini, Qwen, Doubao, or similar) configured via environment variables; support

Supported platforms:

Vision-driven Android automation from screenshots, no DOM access required. \n \n Operates entirely from device screenshots using AI visual understanding; interacts with any visible UI element regardless of underlying technology stack \n Supports taps, swipes, text input, app launches, and complex multi-step interactions via natural language commands \n Requires pre-configured vision model (Gemini, Qwen, Doubao, or similar) with API credentials in environment variables \n Commands run synchronous

Vision-driven browser automation from screenshots, no DOM access required. \n \n Operates in a headless Puppeteer browser that persists across CLI calls, allowing sequential commands without session loss \n Interacts with all visible page elements using natural language prompts; no CSS selectors or accessibility labels needed \n Requires configuration of a vision-capable model (Gemini, Qwen, Doubao, or similar) via environment variables for visual grounding \n Supports connect, take_screenshot,