LLM SLO Engineering: Defining Reliability for AI Systems

LLM quality is probabilistic, not deterministic.

SLOs must capture both system performance and output quality.

Include latency, accuracy, and safety metrics.

Use weighted targets for different user tiers.

Allocate error budgets across quality and latency.

Use error budgets to decide when to slow releases.

Track SLO compliance in dashboards.

Report trends to stakeholders regularly.

Define playbooks for SLO breaches.

Use rollbacks or routing changes to restore targets.

Separate user-facing SLOs from internal engineering indicators.

Define SLOs per workflow when accuracy expectations differ.

Include refusal correctness for safety-sensitive products.

Track coverage rates for retrieval-driven systems.

Set tiered SLOs for premium versus free users.

Keep dependency SLOs visible so upstream issues are not hidden.

Define acceptable degradation modes for peak traffic.

Document SLO owners so accountability is clear.

Use representative evaluation sets that reflect production traffic.

Track factuality, consistency, and safety alongside accuracy.

Measure drift when models or prompts change.

Include latency impact in quality trade-off decisions.

Use human review for high-impact quality metrics.

Segment metrics by region and language to spot gaps.

Define success thresholds per feature to avoid one-size targets.

Review SLOs quarterly to keep them aligned with product goals.

Split error budgets across latency, quality, and safety dimensions.

Define burn rates to decide when to slow releases.

Use budget exhaustion to trigger incident response steps.

Track budget usage per feature to prevent hidden regressions.

Set separate budgets for experimental features.

Include retriever and tool errors in error accounting.

Align error budgets with customer SLAs where applicable.

Publish budget status for cross-team visibility.

Run monthly SLO reviews to validate targets and metrics.

Document changes so historical performance remains comparable.

Include product stakeholders to align on trade-offs.

Use post-incident reviews to refine SLO definitions.

Track SLO misses by root cause to guide investment.

Add guardrails for data quality changes that affect SLOs.

Refresh evaluation sets to avoid outdated benchmarks.

Keep a backlog of SLO improvements with owners and timelines.

Do I need separate SLOs per feature? Yes for high-impact workflows.

What is the fastest win? Start with latency and refusal rate targets.

What is the biggest risk? Overly strict SLOs that slow innovation.