Vision Framework Computer Vision

Guides you through implementing computer vision: subject segmentation, hand/body pose detection, person detection, text recognition, barcode detection, document scanning, and combining Vision APIs to solve complex problems.

When to Use This Skill

Use when you need to:

• ☑ Isolate subjects from backgrounds (subject lifting)
• ☑ Detect and track hand poses for gestures
• ☑ Detect and track body poses for fitness/action classification
• ☑ Segment multiple people separately
• ☑ Exclude hands from object bounding boxes (combining APIs)
• ☑ Choose between VisionKit and Vision framework
• ☑ Combine Vision with CoreImage for compositing
• ☑ Decide which Vision API solves your problem
• ☑ Recognize text in images (OCR)
• ☑ Detect barcodes and QR codes
• ☑ Scan documents with perspective correction
• ☑ Extract structured data from documents (iOS 26+)
• ☑ Build live scanning experiences (DataScannerViewController)

Example Prompts

"How do I isolate a subject from the background?" "I need to detect hand gestures like pinch" "How can I get a bounding box around an object without including the hand holding it?" "Should I use VisionKit or Vision framework for subject lifting?" "How do I segment multiple people separately?" "I need to detect body poses for a fitness app" "How do I preserve HDR when compositing subjects on new backgrounds?" "How do I recognize text in an image?" "I need to scan QR codes from camera" "How do I extract data from a receipt?" "Should I use DataScannerViewController or Vision directly?" "How do I scan documents and correct perspective?" "I need to extract table data from a document"

Red Flags

Signs you're making this harder than it needs to be:

• ❌ Manually implementing subject segmentation with CoreML models
• ❌ Using ARKit just for body pose (Vision works offline)
• ❌ Writing gesture recognition from scratch (use hand pose + simple distance checks)
• ❌ Processing on main thread (blocks UI - Vision is resource intensive)
• ❌ Training custom models when Vision APIs already exist
• ❌ Not checking confidence scores (low confidence = unreliable landmarks)
• ❌ Forgetting to convert coordinates (lower-left origin vs UIKit top-left)
• ❌ Building custom text recognizer when VNRecognizeTextRequest exists
• ❌ Using AVFoundation + Vision when DataScannerViewController suffices
• ❌ Processing every camera frame for scanning (skip frames, use region of interest)
• ❌ Enabling all barcode symbologies when you only need one (performance hit)
• ❌ Ignoring RecognizeDocumentsRequest when you need table/list structure (iOS 26+)

Mandatory First Steps

Before implementing any Vision feature:

1. Choose the Right API (Decision Tree)

What do you need to do?

┌─ Isolate subject(s) from background?
│  ├─ Need system UI + out-of-process → VisionKit
│  │  └─ ImageAnalysisInteraction (iOS/iPadOS)
│  │  └─ ImageAnalysisOverlayView (macOS)
│  ├─ Need custom pipeline / HDR / large images → Vision
│  │  └─ VNGenerateForegroundInstanceMaskRequest
│  └─ Need to EXCLUDE hands from object → Combine APIs
│     └─ Subject mask + Hand pose + custom masking (see Pattern 1)
│
├─ Segment people?
│  ├─ All people in one mask → VNGeneratePersonSegmentationRequest
│  └─ Separate mask per person (up to 4) → VNGeneratePersonInstanceMaskRequest
│
├─ Detect hand pose/gestures?
│  ├─ Just hand location → VNDetectHumanRectanglesRequest
│  └─ 21 hand landmarks → VNDetectHumanHandPoseRequest
│     └─ Gesture recognition → Hand pose + distance checks
│
├─ Detect body pose?
│  ├─ 2D normalized landmarks → VNDetectHumanBodyPoseRequest
│  ├─ 3D real-world coordinates → VNDetectHumanBodyPose3DRequest
│  └─ Action classification → Body pose + CreateML model
│
├─ Face detection?
│  ├─ Just bounding boxes → VNDetectFaceRectanglesRequest
│  └─ Detailed landmarks → VNDetectFaceLandmarksRequest
│
├─ Person detection (location only)?
│  └─ VNDetectHumanRectanglesRequest
│
├─ Recognize text in images?
│  ├─ Real-time from camera + need UI → DataScannerViewController (iOS 16+)
│  ├─ Processing captured image → VNRecognizeTextRequest
│  │  ├─ Need speed (real-time camera) → recognitionLevel = .fast
│  │  └─ Need accuracy (documents) → recognitionLevel = .accurate
│  └─ Need structured documents (iOS 26+) → RecognizeDocumentsRequest
│
├─ Detect barcodes/QR codes?
│  ├─ Real-time camera + need UI → DataScannerViewController (iOS 16+)
│  └─ Processing image → VNDetectBarcodesRequest
│
└─ Scan documents?
   ├─ Need built-in UI + perspective correction → VNDocumentCameraViewController
   ├─ Need structured data (tables, lists) → RecognizeDocumentsRequest (iOS 26+)
   └─ Custom pipeline → VNDetectDocumentSegmentationRequest + perspective correction

2. Set Up Background Processing

NEVER run Vision on main thread:

let processingQueue = DispatchQueue(label: "com.yourapp.vision", qos: .userInitiated)

processingQueue.async {
    do {
        let request = VNGenerateForegroundInstanceMaskRequest()
        let handler = VNImageRequestHandler(cgImage: image)
        try handler.perform([request])

        // Process observations...

        DispatchQueue.main.async {
            // Update UI
        }
    } catch {
        // Handle error
    }
}

3. Choose the Right Request Handler

Processing video frames? Use VNSequenceRequestHandler (maintains inter-frame state for temporal smoothing). For single images, use VNImageRequestHandler. Creating a new VNImageRequestHandler per frame discards temporal context and causes jittery results. See axiom-vision-ref for full comparison and code examples.

4. Verify Platform Availability

Common Patterns

Pattern 1: Isolate Object While Excluding Hand

User's original problem: Getting a bounding box around an object held in hand, without including the hand.

Root cause: VNGenerateForegroundInstanceMaskRequest is class-agnostic and treats hand+object as one subject.

Solution: Combine subject mask with hand pose to create exclusion mask.

// 1. Get subject instance mask
let subjectRequest = VNGenerateForegroundInstanceMaskRequest()
let handler = VNImageRequestHandler(cgImage: sourceImage)
try handler.perform([subjectRequest])

guard let subjectObservation = subjectRequest.results?.first as? VNInstanceMaskObservation else {
    fatalError("No subject detected")
}

// 2. Get hand pose landmarks
let handRequest = VNDetectHumanHandPoseRequest()
handRequest.maximumHandCount = 2
try handler.perform([handRequest])

guard let handObservation = handRequest.results?.first as? VNHumanHandPoseObservation else {
    // No hand detected - use full subject mask
    let mask = try subjectObservation.createScaledMask(
        for: subjectObservation.allInstances,
        croppedToInstancesContent: false
    )
    return mask
}

// 3. Create hand exclusion region from landmarks
let handPoints = try handObservation.recognizedPoints(.all)
let handBounds = calculateConvexHull(from: handPoints)  // Your implementation

// 4. Subtract hand region from subject mask using CoreImage
let subjectMask = try subjectObservation.createScaledMask(
    for: subjectObservation.allInstances,
    croppedToInstancesContent: false
)

let subjectCIMask = CIImage(cvPixelBuffer: subjectMask)
let handMask = createMaskFromRegion(handBounds, size: sourceImage.size)
let finalMask = subtractMasks(handMask: handMask, from: subjectCIMask)

// 5. Calculate bounding box from final mask
let objectBounds = calculateBoundingBox(from: finalMask)

Helper: Convex Hull

func calculateConvexHull(from points: [VNRecognizedPointKey: VNRecognizedPoint]) -> CGRect {
    // Get high-confidence points
    let validPoints = points.values.filter { $0.confidence > 0.5 }

    guard !validPoints.isEmpty else { return .zero }

    // Simple bounding rect (for more accuracy, use actual convex hull algorithm)
    let xs = validPoints.map { $0.location.x }
    let ys = validPoints.map { $0.location.y }

    let minX = xs.min()!
    let maxX = xs.max()!
    let minY = ys.min()!
    let maxY = ys.max()!

    return CGRect(
        x: minX,
        y: minY,
        width: maxX 
how to use axiom-vision
CursorClaude CodeClineCodexWindsurfGooseGitHub CopilotVS CodeGemini CLIKiloOpencodeKimi CLIRoo ClineAiderContinueZedBolt.newLovablev0Replit AgentSweepSmol DeveloperDevinCavemanAmpAntigravity
How to use axiom-vision on Cursor
AI-first code editor with Composer
1
Prerequisites
Before installing skills in Cursor, ensure your development environment meets these requirements:
›Cursor installed and configured on your development machine
›Node.js version 16.0+ with npm package manager (verify with node --version)
›Active project directory or workspace where you want to add axiom-vision
2
Execute installation command
Execute the skills CLI command in your project's root directory to begin installation:
$npx skills add https://github.com/charleswiltgen/axiom --skill axiom-visioncopy
The skills CLI fetches axiom-vision from GitHub repository charleswiltgen/axiom and configures it for Cursor.
3
Select Cursor when prompted
The CLI will show a list of available agents. Use arrow keys to navigate and space to select Cursor:
◆ Which agents do you want to install to?
│
│ ── Universal (.agents/skills) ── always included ────
│   • Amp
│   • Antigravity
│   • Cline
│   • Codex
│ ●Cursor(selected)
│   • Cursor
│   • Windsurf
4
Verify installation
Confirm successful installation by checking the skill directory location:
.cursor/skills/axiom-vision
Reload or restart Cursor to activate axiom-vision. Access the skill through slash commands (e.g., /axiom-vision) or your agent's skill management interface.
⚠
Security & Verification Notice
We perform automated surface-level scans (Gen AI Scanner, Socket, Snyk) during installation. These checks detect common vulnerabilities but do not guarantee complete security. Always review skill source code and verify the publisher's reputation before production use.
Skills execute code in your development environment. Always verify the publisher's identity, review recent commits, and test in isolated environments before production deployment.
Additional Resources
›View source code on GitHub›Skills CLI documentation›Learn more about Cursor›What are agent skills?