Foundation Models Diagnostics

Overview

Foundation Models issues manifest as context window exceeded errors, guardrail violations, slow generation, availability failures, and unexpected output. Core principle 80% of Foundation Models problems stem from misunderstanding model capabilities (3B parameter device-scale model, not world knowledge), context limits (4096 tokens), or availability requirements—not framework bugs.

Red Flags — Suspect Foundation Models Issue

If you see ANY of these, suspect a Foundation Models misunderstanding, not framework breakage:

• Generation takes >5 seconds
• Error: exceededContextWindowSize
• Error: guardrailViolation
• Error: unsupportedLanguageOrLocale
• Model gives hallucinated/wrong output
• UI freezes during generation
• Feature works in simulator but not on device
• ❌ FORBIDDEN "Foundation Models is broken, we need a different AI"
  • Foundation Models powers Apple Intelligence across millions of devices
  • Wrong output = wrong use case (world knowledge vs summarization)
  • Do not rationalize away the issue—diagnose it

Critical distinction Foundation Models is a device-scale model (3B parameters) optimized for summarization, extraction, classification—NOT world knowledge or complex reasoning. Using it for the wrong task guarantees poor results.

Mandatory First Steps

ALWAYS run these FIRST (before changing code):

// 1. Check availability
let availability = SystemLanguageModel.default.availability

switch availability {
case .available:
    print("✅ Available")
case .unavailable(let reason):
    print("❌ Unavailable: \(reason)")
    // Possible reasons:
    // - Device not Apple Intelligence-capable
    // - Region not supported
    // - User not opted in
}

// Record: "Available? Yes/no, reason if not"

// 2. Check supported languages
let supported = SystemLanguageModel.default.supportedLanguages
print("Supported languages: \(supported)")
print("Current locale: \(Locale.current.language)")

if !supported.contains(Locale.current.language) {
    print("⚠️ Current language not supported!")
}

// Record: "Language supported? Yes/no"

// 3. Check context usage
let session = LanguageModelSession()
// After some interactions:
print("Transcript entries: \(session.transcript.entries.count)")

// Rough estimation (not exact):
let transcriptText = session.transcript.entries
    .map { $0.content }
    .joined()
print("Approximate chars: \(transcriptText.count)")
print("Rough token estimate: \(transcriptText.count / 3)")
// 4096 token limit ≈ 12,000 characters

// Record: "Approaching context limit? Yes/no"

// 4. Profile with Instruments
// Run with Foundation Models Instrument template
// Check:
// - Initial model load time
// - Token counts (input/output)
// - Generation time per request
// - Areas for optimization

// Record: "Latency profile: [numbers from Instruments]"

// 5. Inspect transcript for debugging
print("Full transcript:")
for entry in session.transcript.entries {
    print("Entry: \(entry.content.prefix(100))...")
}

// Record: "Any unusual entries? Repeated content?"

What this tells you

• Unavailable → Proceed to Pattern 1a/1b/1c (availability issues)
• Context exceeded → Proceed to Pattern 2a (token limit)
• Guardrail error → Proceed to Pattern 2b (content policy)
• Language error → Proceed to Pattern 2c (unsupported language)
• Wrong output → Proceed to Pattern 3a/3b/3c (output quality)
• Slow generation → Proceed to Pattern 4a/4b/4c/4d (performance)
• UI frozen → Proceed to Pattern 5a (main thread blocking)

MANDATORY INTERPRETATION

Before changing ANY code, identify ONE of these:

1. If availability = .unavailable → Device/region/opt-in issue (not code bug)
2. If error is exceededContextWindowSize → Too many tokens (condense transcript)
3. If error is guardrailViolation → Content policy triggered (not model failure)
4. If error is unsupportedLanguageOrLocale → Language not supported (check supported list)
5. If output is hallucinated → Wrong use case (world knowledge vs extraction)
6. If generation >5 seconds → Not streaming or need optimization
7. If UI frozen → Calling on main thread (use Task {})

If diagnostics are contradictory or unclear

• STOP. Do NOT proceed to patterns yet
• Add detailed logging to every respond() call
• Run with Instruments Foundation Models template
• Establish baseline: what's actually happening vs what you assumed

Decision Tree

Foundation Models problem?
│
├─ Won't start?
│  ├─ .unavailable → Availability issue
│  │  ├─ Device not capable? → Pattern 1a (device requirement)
│  │  ├─ Region restriction? → Pattern 1b (regional availability)
│  │  └─ User not opted in? → Pattern 1c (Settings check)
│  │
├─ Generation fails?
│  ├─ exceededContextWindowSize → Context limit
│  │  └─ Long conversation or verbose prompts? → Pattern 2a (condense)
│  │
│  ├─ guardrailViolation → Content policy
│  │  └─ Sensitive or inappropriate content? → Pattern 2b (handle gracefully)
│  │
│  ├─ unsupportedLanguageOrLocale → Language issue
│  │  └─ Non-English or unsupported language? → Pattern 2c (language check)
│  │
│  └─ Other error → General error handling
│     └─ Unknown error type? → Pattern 2d (catch-all)
│
├─ Output wrong?
│  ├─ Hallucinated facts → Wrong model use
│  │  └─ Asking for world knowledge? → Pattern 3a (use case mismatch)
│  │
│  ├─ Wrong structure → Parsing issue
│  │  └─ Manual JSON parsing? → Pattern 3b (use @Generable)
│  │
│  ├─ Missing data → Tool needed
│  │  └─ Need external information? → Pattern 3c (tool calling)
│  │
│  └─ Inconsistent output → Sampling issue
│     └─ Different results each time? → Pattern 3d (temperature/greedy)
│
├─ Too slow?
│  ├─ Initial delay (1-2s) → Model loading
│  │  └─ First request slow? → Pattern 4a (prewarm)
│  │
│  ├─ Long wait for results → Not streaming
│  │  └─ User waits 3-5s? → Pattern 4b (streaming)
│  │
│  ├─ Verbose schema → Token overhead
│  │  └─ Large @Generable type? → Pattern 4c (includeSchemaInPrompt)
│  │
│  └─ Complex prompt → Too much processing
│     └─ Massive prompt or task? → Pattern 4d (break down)
│
└─ UI frozen?
   └─ Main thread blocked → Async issue
      └─ App unresponsive during generation? → Pattern 5a (Task {})

Diagnostic Patterns

Pattern 1a: Device Not Capable

Symptom:

• SystemLanguageModel.default.availability = .unavailable
• Reason: Device not Apple Intelligence-capable

Diagnosis:

let availability = SystemLanguageModel.default.availability

switch availability {
case .available:
    print("✅ Available")
case .unavailable(let reason):
    print("❌ Reason: \(reason)")
    // Check if device-related
}

Fix:

// ❌ BAD - No availability UI
let session = LanguageModelSession() // Crashes on unsupported devices

// ✅ GOOD - Graceful UI
struct AIFeatureView: View {
    @State private var availability = SystemLanguageModel.default.availability

    var body: some View {
        switch availability {
        case .available:
            AIContentView()
        case .unavailable:
            VStack {
                Image(systemName: "cpu")
                Text("AI features require Apple Intelligence")
                    .font(.headline)
                Text("Available on iPhone 15 Pro and later")
                    .font(.caption)
                    .foregroundColor(.secondary)
            }
        }
    }
}

Time cost: 5-10 minutes to add UI

P