SKILL: Week 4: Crash Analysis and Exploitability Assessment

Metadata

• Skill Name: crash-analysis
• Folder: offensive-crash-analysis
• Source: https://github.com/SnailSploit/offensive-checklist/blob/main/4-crash-analysis.md

Description

Week 4 exploit development curriculum. Crash triage and analysis methodology: WinDbg/GDB analysis, ASAN/MSAN output interpretation, exploitability assessment, register/stack trace reading, root cause identification. Use when analyzing crash dumps, assessing exploitability, or understanding fuzzer-generated crashes.

Trigger Phrases

Use this skill when the conversation involves any of: crash analysis, crash triage, WinDbg, GDB, ASAN, MSAN, exploitability, stack trace, register dump, segfault, null deref, access violation, week 4

Instructions for Claude

When this skill is active:

1. Load and apply the full methodology below as your operational checklist
2. Follow steps in order unless the user specifies otherwise
3. For each technique, consider applicability to the current target/context
4. Track which checklist items have been completed
5. Suggest next steps based on findings

---

Full Methodology

Week 4: Crash Analysis and Exploitability Assessment

Overview

created by AnotherOne from @Pwn3rzs Telegram channel.

After finding potential vulnerabilities through fuzzing (Week 2) or patch diffing (Week 3), the next critical step is analyzing crashes to determine if they're exploitable. This week focuses on crash triage, debugger mastery, and techniques for identifying how to reach vulnerable code paths from attacker-controlled input.

Once you've confirmed a crash is exploitable and built a PoC, you'll be ready for Basic Exploitation in Week 5.

Prerequisites

Before starting this week, ensure you have:

• A Windows VM (for WinDbg labs) and a Linux VM (for GDB/ASAN/CASR labs).
• Completed Week 2 fuzzing labs, including running AFL++ or libFuzzer against at least one C/C++ target
• Completed (or skimmed) Week 3 patch diffing labs:
  • Familiar with Ghidriff/Diaphora diff reports and how to interpret changed functions
  • Understand how to extract Windows updates and Linux kernel patches
  • Reviewed at least one case study (CVE-2022-34718 EvilESP, CVE-2024-1086 nf_tables, or 7-Zip symlink bugs)
• Comfortable understanding from Week 1 of basic vulnerability classes (buffer overflow, UAF, integer bugs, info leaks) and their exploit primitives

Crash Analysis Decision Tree

Use this decision tree to select the appropriate tools and workflow for any crash you encounter:

┌─────────────────────────────────────────────────────────────────────┐
│                        CRASH RECEIVED                               │
└─────────────────────────────────────────────────────────────────────┘
                                │
                                ▼
                    ┌───────────────────────┐
                    │ Source code available?│
                    └───────────────────────┘
                      │                    │
                     Yes                   No
                      │                    │
                      ▼                    ▼
        ┌─────────────────────┐   ┌──────────────────────────┐
        │ Recompile with      │   │ What platform?           │
        │ ASAN + UBSAN        │   └──────────────────────────┘
        │ (Day 2)             │     │         │         │
        └─────────────────────┘     │         │         │
                      │          Windows   Linux    Mobile
                      │             │         │         │
                      ▼             ▼         ▼         ▼
        ┌─────────────────────┐ ┌───────┐ ┌───────┐ ┌───────────┐
        │ Run crash input     │ │WinDbg │ │Pwndbg │ │ Tombstone │
        │ Get detailed report │ │+ TTD  │ │+ rr   │ │ + Frida   │
        └─────────────────────┘ │(Day 1)│ │(Day 1)│ │ (Future)  │
                      │         └───────┘ └───────┘ └───────────┘
                      │             │         │         │
                      └─────────────┴────┬────┴─────────┘
                                         │
                                         ▼
                    ┌─────────────────────────────────────┐
                    │ Crash requires special environment? │
                    └─────────────────────────────────────┘
                       │                              │
                      Yes                             No
                       │                              │
                       ▼                              │
        ┌─────────────────────────────┐               │
        │ Setup reproduction env:     │               │
        │ - Network (tcpdump, proxy)  │               │
        │ - Files (strace, procmon)   │               │
        │ - Services (docker, VM)     │               │
        └─────────────────────────────┘               │
                       │                              │
                       └──────────────┬───────────────┘
                                      │
                                      ▼
                            ┌─────────────────────┐
                            │ Crash type known?   │
                            └─────────────────────┘
                              │                 │
                             Yes                No
                              │                 │
                              ▼                 ▼
                ┌─────────────────────┐  ┌─────────────────────┐
                │ Run CASR for        │  │ Manual analysis:    │
                │ classification      │  │ - Examine registers │
                │ (Day 3)             │  │ - Check memory      │
                └─────────────────────┘  │ - Disassemble       │
                              │          │ (Day 3)             │
                              │          └─────────────────────┘
                              │                 │
                              └────────┬────────┘
                                       │
                                       ▼
                          ┌─────────────────────────┐
                          │ EXPLOITABILITY ASSESS   │
                          │ - Check mitigations     │
                          │ - Control analysis      │
                          │ - Reachability (Day 4)  │
                          └─────────────────────────┘
                                       │
                                       ▼
                          ┌─────────────────────────┐
                          │ Multiple crashes?       │
                          └─────────────────────────┘
                            │                    │
                           Yes                   No
                            │                    │
                            ▼                    ▼
              ┌─────────────────────┐   ┌─────────────────────┐
              │ Deduplicate (Day 5) │   │ Minimize (Day 5)    │
              │ - CASR cluster      │   │ - afl-tmin          │
              │ - Stack hash        │   │ - Manual reduction  │
              └─────────────────────┘   └─────────────────────┘
                            │                    │
                            └────────┬───────────┘
                                     │
                                     ▼
                        ┌─────────────────────────┐
                        │ Create PoC (Day 6)      │
                        │ - Python + pwntools     │
                        │ - Verify reliability    │
                        │ - Document findings     │
                        └─────────────────────────┘

Quick Reference - Tool Selection by Scenario:

Day 1: Debugger Fundamentals and Crash Dump Analysis

• Goal: Learn Windows Debugger (WinDbg) and Linux debugger (GDB + Pwndbg) for analyzing application crashes.
• Activities:
  • Reading:
    • "Practical Malware Analysis" by Michael Sikorski - Chapter 9 and 10
    • WinDbg Official Documentation
    • Pwndbg Documentation
  • Online Resources:
    • Common WinDbg Commands
    • Debugging Tools for Windows
    • GDB Quick Reference
  • Tool Setup:
    • Windows: Install WinDbg Preview from Microsoft Store
    • Linux: Install GDB with Pwndbg enhancement
    • Install Windows SDK for symbol support
  • Exercise:
    • Analyze 5 pre-generated crash dumps (Windows and Linux)
    • Identify crash type and root cause for each

Reproduction Fidelity

[!IMPORTANT] Before any crash analysis, ensure you can reproduce the crash reliably. A crash that only happens "sometimes" or "on the fuzzer's machine" is nearly impossible to analyze or exploit. This section establishes the mandatory checklist for achieving reproduction fidelity.

Reproduction Fidelity Checklist

Before analyzing any crash, verify these match between discovery and analysis environments:

┌─────────────────────────────────────────────────────────────────┐
│ REPRODUCTION FIDELITY CHECKLIST                                 │
├─────────────────────────────────────────────────────────────────┤
│ System Environment                                              │
│ [ ] OS/Kernel version     : ________________________________    │
│ [ ] libc version          : ________________________________    │
│ [ ] CPU architecture      : [ ] x86 [ ] x86_64 [ ] ARM64        │
│ [ ] Container/VM          : [ ] Native [ ] Docker [ ] VM        │
│ [ ] ASLR state            : [ ] Enabled [ ] Disabled            │
├─────────────────────────────────────────────────────────────────┤
│ Process Environment                                             │
│ [ ] argv (command-line)   : ________________________________    │
│ [ ] Environment variables : ________________________________    │
│ [ ] Working directory     : ________________________________    │
│ [ ] Locale (LC_ALL, LANG) : ________________________________    │
│ [ ] umask / permissions   : ________________________________    │
├─────────────────────────────────────────────────────────────────┤
│ Input Path                                                      │
│ [ ] Input source          : [ ] stdin [ ] file [ ] network      │
│ [ ] Input file path       : ________________________________    │
│ [ ] Network port/protocol : ________________________________    │
├─────────────────────────────────────────────────────────────────┤
│ Build Configuration                                             │
│ [ ] Compiler version      : ________________________________    │
│ [ ] Optimization level    : [ ] -O0 [ ] -O1 [ ] -O2 [ ] -O3     │
│ [ ] Sanitizers            : [ ] ASAN [ ] UBSAN [ ] TSAN [ ] None│
│ [ ] Debug symbols         : [ ] Yes [ ] No                      │
│ [ ] Mitigations           : [ ] PIE [ ] Canary [ ] RELRO        │
└─────────────────────────────────────────────────────────────────┘

Essential Environment Knobs

ASAN/UBSAN Options (Linux/macOS):

# Full ASAN options for crash analysis
export ASAN_OPTIONS="\
abort_on_error=1:\
symbolize=1:\
detect_leaks=1:\
disable_coredump=0:\
halt_on_error=1:\
print_stats=1:\
check_initialization_order=1:\
detect_stack_use_after_return=1:\
quarantine_size_mb=256"

# UBSAN options
export UBSAN_OPTIONS="\
print_stacktrace=1:\
halt_on_error=1:\
suppressions=ubsan_suppressions.txt"

# Symbolizer path (required for readable stack traces)
export ASAN_SYMBOLIZER_PATH=$(command -v llvm-symbolizer)

glibc Allocator Tuning (Linux):

# Enable glibc heap consistency checks (catch corruption early)
export MALLOC_CHECK_=3

# Modern glibc tunable interface (glibc 2.26+)
export GLIBC_TUNABLES="\
glibc.malloc.check=3:\
glibc.malloc.perturb=165"

# What these do:
# MALLOC_CHECK_=3: Abort on heap corruption detection
# glibc.malloc.perturb=165: Fill freed memory with 0xA5 (helps detect UAF)

Core Dump Configuration (Linux):

# Enable unlimited core dumps
ulimit -c unlimited

# Verify core pattern (where dumps go)
cat /proc/sys/kernel/core_pattern

# For local dumps in CWD (temporary, affects system):
# echo 'core.%e.%p' | sudo tee /proc/sys/kernel/core_pattern

ASLR Control (Linux - for deterministic analysis):

# Check current ASLR state
cat /proc/sys/kernel/randomize_va_space
# 0 = disabled, 1 = conservative, 2 = full

# Disable ASLR for current shell (temporary, per-process)
setarch $(uname -m) -R ./target < crash_input

# Or system-wide (DANGEROUS - only for isolated VMs):
# echo 0 | sudo tee /proc/sys/kernel/randomize_va_space

Input Path Matching

The crash may behave differently depending on HOW input reaches the target:

# If fuzzer used stdin:
./target < crash_input

# If fuzzer used file argument:
./target crash_input

# If fuzzer used network:
cat crash_input | nc localhost 8080

# WRONG: Mixing input paths can change behavior!
# Fuzzer: ./target @@ (file)
# You:    ./target < crash (stdin)  # May not reproduce!

Example: stdin vs file difference:

// Some programs behave differently:
// - stdin may be line-buffered
// - File may be memory-mapped
// - Network may have different read chunk sizes

// This can affect:
// - Buffer contents at crash time
// - Heap layout (different allocation patterns)
// - Race conditions (timing changes)

Quick Reproduction Test Script

#!/bin/bash
# repro_test.sh - Verify crash reproduction

CRASH_INPUT="$1"
TARGET="$2"
EXPECTED_SIGNAL="${3:-11}"  # Default: SIGSEGV (11)

echo "[*] Testing reproduction of $(basename $CRASH_INPUT)"
echo "[*] Target: $TARGET"
echo "[*] Expected signal: $EXPECTED_SIGNAL"

# Set up environment
ulimit -c unlimited
export ASAN_OPTIONS="abort_on_error=1:symbolize=1"

# Run 10 times
CRASHES=0
for i in {1..10}; do
    timeout 5s $TARGET < "$CRASH_INPUT" 2>/dev/null
    EXIT_CODE=$?

    # Check for crash signal (128 + signal number)
    if [ $EXIT_CODE -gt 128 ]; then
        SIGNAL=$((EXIT_CODE - 128))
        if [ $SIGNAL -eq $EXPECTED_SIGNAL ] || [ $SIGNAL -eq 6 ]; then
            ((CRASHES++))
        fi
    fi
done

echo "[*] Crash rate: $CRASHES/10"
if [ $CRASHES -ge 9 ]; then
    echo "[+] Reproduction: RELIABLE"
elif [ $CRASHES -ge 5 ]; then
    echo "[!] Reproduction: FLAKY - investigate environment"
else
    echo "[-] Reproduction: FAILED - check environment checklist"
fi

Installing WinDbg and Symbol Support

WinDbg Preview (recommended - modern UI):

winget install Microsoft.WinDbg

Windows SDK Debugging Tools (includes cdb.exe for command-line/batch analysis):

# Option 1: Install via winget (Windows SDK)
winget install --source winget --exact --id Microsoft.WindowsSDK.10.0.26100

# Option 2: Download from Microsoft
# https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/
# During installation, select "Debugging Tools for Windows"

# After installation, cdb.exe is located at:
# C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe

# Add to PATH for convenience (run as Administrator):
setx PATH "%PATH%;C:\Program Files (x86)\Windows Kits\10\Debuggers\x64" /M

# Or use full path in scripts:
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -z dump.dmp -c "!analyze -v; q"

Configure Symbol Path:

# In WinDbg Settings -> Default Symbol Path, or:
# In WinDbg command window:
.sympath SRV*C:\Symbols*https://msdl.microsoft.com/download/symbols

# Or set environment variable permanently (recommended):
setx _NT_SYMBOL_PATH "SRV*C:\Symbols*https://msdl.microsoft.com/download/symbols"

# Create symbols cache directory
mkdir C:\Symbols

# Reload symbols (in debugger)
.reload /f

Linux Crash Dump Generation and Pwndbg Setup

[!HINT] While Windows uses WinDbg, Linux crash analysis uses GDB enhanced with Pwndbg. This section covers parallel Linux setup.

Installing Pwndbg:

# Install GDB
sudo apt install gdb

# Install Pwndbg (recommended for crash analysis)
cd ~/tools
git clone --depth 1 https://github.com/pwndbg/pwndbg
cd pwndbg
./setup.sh

# Verify installation
gdb -q -ex "quit" 2>&1 | grep -q "pwndbg" && echo "pwndbg installed successfully"

[!WARNING] Pwndbg is installed per-user in ~/.gdbinit. If you run sudo gdb, it uses root's home directory and won't find your pwndbg config. Solutions: For crash analysis of your own compiled test programs, you typically don't need sudo. Only use sudo when attaching to system processes or analyzing setuid binaries.

# Option 1: Use gdb as regular user (recommended for most analysis)
cd ~/crash_analysis_lab
gdb ./vuln_no_protect -c core.dump

# Option 2: If you MUST use sudo (e.g., attaching to privileged process)
sudo -E gdb ./program  # -E preserves your environment including HOME

# Option 3: Install pwndbg for root as well
sudo su -
cd /root
git clone https://github.com/pwndbg/pwndbg
cd pwndbg && ./setup.sh
exit

# Option 4: Explicitly source pwndbg in sudo gdb session
sudo gdb -ex "source /home/<YOUR_USER>/tools/pwndbg/gdbinit.py" ./program

Configuring Core Dumps on Linux:

# Check current core dump configuration
cat /proc/sys/kernel/core_pattern

# Enable core dumps for current shell (recommended for learning)
ulimit -c unlimited

[!TIP] For the exercises in this course, you typically only need:

ulimit -c unlimited  # In your current shell

On modern Ubuntu/Debian with systemd, cores are handled by systemd-coredump even if you set ulimit. Use coredumpctl to list and debug them.

[!WARNING] Optional: Local core files in CWD (modifies system-wide settings)

If you specifically need core files in your working directory instead of systemd-coredump:

# This is SYSTEM-WIDE and may interfere with other tooling
echo 'core.%e.%p' | sudo tee /proc/sys/kernel/core_pattern

Additional kernel settings that affect core dumps:

• kernel.core_uses_pid: Append PID to core filename
• fs.suid_dumpable: Controls dumps for setuid binaries (0=disabled, 1=enabled, 2=suidsafe)

Building a Vulnerable Test Suite for Linux

Create these vulnerable C programs to generate real crashes:

# Create a directory for crash analysis practice
mkdir -p ~/crash_analysis_lab/{src,crashes,cores}
cd ~/crash_analysis_lab/src

vulnerable_suite.c - Save this file for testing multiple vulnerability types:

// ~/crash_analysis_lab/src/vulnerable_suite.c - Compile with different flags for different exercises
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

// 1. Stack Buffer Overflow
void stack_overflow(char *input) {
    char buffer[64];
    printf("[*] Copying input to 64-byte buffer...\n");
    strcpy(buffer, input);  // No bounds check!
    printf("[*] Buffer: %s\n", buffer);
}

// 2. Heap Buffer Overflow
void heap_overflow(char *input) {
    char *buf = malloc(32);
    printf("[*] Allocated 32 bytes at %p\n", buf);
    strcpy(buf, input);  // Overflow heap buffer
    printf("[*] Buffer: %s\n", buf);
    free(buf);
}

// 3. Use-After-Free
void use_after_free() {
    char *ptr = malloc(64);
    strcpy(ptr, "Hello, World!");
    printf("[*] Allocated at %p: %s\n", ptr, ptr);
    free(ptr);
    printf("[*] Freed, now accessing...\n");
    printf("[*] UAF read: %s\n", ptr);  // UAF read - may print stale data
    ptr[0] = 'X';  // UAF write - may corrupt allocator state
}

// 4. Double Free
void double_free() {
    char *ptr = malloc(64);
    printf("[*] Allocated at %p\n", ptr);
    free(ptr);
    printf("[*] First free done\n");
    free(ptr);  // Double free!
}

// 5. NULL Pointer Dereference
void null_deref(int trigger) {
    char *ptr = trigger ? malloc(10) : NULL;
    printf("[*] ptr = %p\n", ptr);
    *ptr = 'A';  // NULL deref if trigger is 0
}

void print_usage(char *prog) {
    printf("Usage: %s <test_num> [input]\n", prog);
    printf("Tests:\n");
    printf("  1 <input>  - Stack overflow (need ~100+ chars)\n");
    printf("  2 <input>  - Heap overflow (need ~50+ chars)\n");
    printf("  3          - Use-after-free\n");
    printf("  4          - Double free\n");
    printf("  5 <0|1>    - NULL deref (0=crash)\n");
    printf("\nExample: %s 1 $(python3 -c \"print('A'*100)\")\n", prog);
}

int main(int argc, char **argv) {
    if (argc < 2) { print_usage(argv[0]); return 1; }
    int test = atoi(argv[1]);

    switch(test) {
        case 1: if (argc<3) return 1; stack_overflow(argv[2]); break;
        case 2: if (argc<3) return 1; heap_overflow(argv[2]); break;
        case 3: use_after_free(); break;
        case 4: double_free(); break;
        case 5: if (argc<3) return 1; null_deref(atoi(argv[2])); break;
        default: print_usage(argv[0]); return 1;
    }
    return 0;
}

Build the test suite:

cd ~/crash_analysis_lab/src

# 1. Build WITHOUT mitigations (for basic crash analysis)
gcc -g -fno-stack-protector -no-pie -z execstack \
    vulnerable_suite.c -o ../vuln_no_protect

# 2. Build WITH ASAN (for detailed memory error reports)
gcc -g -O1 -fsanitize=address -fno-omit-frame-pointer \
    vulnerable_suite.c -o ../vuln_asan

# 3. Build with standard protections (see how mitigations affect crashes)
gcc -g vulnerable_suite.c -o ../vuln_protected

Generate your first crashes:

cd ~/crash_analysis_lab

# Enable core dumps
ulimit -c unlimited

# Test 1: Stack overflow - generates a core dump
./vuln_no_protect 1 $(python3 -c "print('A'*200)")
# You should see: Segmentation fault (core dumped)
# Check for core file: ls -la core* (if core_pattern writes to CWD) or use coredumpctl (systemd systems) or look at output of `cat /proc/sys/kernel/core_pattern`

# Test 2: Stack overflow with ASAN - detailed report
./vuln_asan 1 $(python3 -c "print('A'*200)") 2>&1 | tee crashes/stack_asan.txt
# ASAN prints detailed overflow information

# Test 3: Use-after-free with ASAN
./vuln_asan 3 2>&1 | tee crashes/uaf_asan.txt

# Test 4: NULL dereference - generates core dump
./vuln_no_protect 5 0

Using coredumpctl (systemd systems):

sudo apt install systemd-coredump
# List recent core dumps
coredumpctl list

# Show details of most recent crash
coredumpctl info

# Debug most recent crash with GDB
coredumpctl debug

# Debug specific crash by PID
coredumpctl debug 12345

# Extract core dump to file for offline analysis
coredumpctl dump -o crash.core

# View where cores are stored
cat /etc/systemd/coredump.conf
# [Cor