Monitoring & Observability

Comprehensive patterns for infrastructure monitoring, LLM observability, and quality drift detection. Each category has individual rule files in rules/ loaded on-demand.

Quick Reference

Total: 12 rules across 4 categories

Quick Start

# Prometheus metrics with RED method
from prometheus_client import Counter, Histogram

http_requests = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
http_duration = Histogram('http_request_duration_seconds', 'Request latency',
    buckets=[0.01, 0.05, 0.1, 0.5, 1, 2, 5])

# Langfuse v4 LLM tracing — semantic as_type + inline scoring
from langfuse import observe, get_client

@observe(as_type="generation", name="analyze_content")
async def analyze_content(content: str):
    get_client().update_current_trace(
        user_id="user_123", session_id="session_abc",
        tags=["production", "orchestkit"],
    )
    result = await llm.generate(content)
    get_client().score_current_span(name="response_quality", value=0.85)
    return result

# PSI drift detection
import numpy as np

psi_score = calculate_psi(baseline_scores, current_scores)
if psi_score >= 0.25:
    alert("Significant quality drift detected!")

Infrastructure Monitoring

Prometheus metrics, Grafana dashboards, and alerting for application health.

LLM Observability

Langfuse-based tracing, cost tracking, and evaluation for LLM applications.

Drift Detection

Statistical and quality drift detection for production LLM systems.

Silent Failures

Detection and alerting for silent failures in LLM agents.

Key Decisions

Detailed Documentation

Related Skills

• defense-in-depth - Layer 8 observability as part of security architecture
• devops-deployment - Observability integration with CI/CD and Kubernetes
• resilience-patterns - Monitoring circuit breakers and failure scenarios
• llm-evaluation - Evaluation patterns that integrate with Langfuse scoring
• caching - Caching strategies that reduce costs tracked by Langfuse