SLI, SLO, SLA và Error Budget — Site Reliability Engineering

SLA (Service Level Agreement)
  ↓ business contract, legal consequences
SLO (Service Level Objective)
  ↓ internal target, drives engineering decisions
SLI (Service Level Indicator)
  ↓ measurement, actual user experience

User journey: Checkout flow
  1. Add to cart     → 99.9% availability, p99 < 500ms
  2. View cart       → 99.5% availability, p99 < 1s
  3. Enter payment   → 99.95% availability, p99 < 300ms  ← Critical
  4. Confirm order   → 99.99% availability, p99 < 200ms  ← Most critical

Service: Payment API
SLA: 99.9% monthly uptime
Measurement: Successful 2xx responses / total requests
Exclusions: Planned maintenance (max 4 hours/month), user errors (4xx)
Consequences:
  - 99.9% - 99.0%: 10% credit
  - 99.0% - 95.0%: 25% credit
  - < 95.0%:       100% credit

Total time = 30 days × 24 hours × 60 min = 43,200 minutes
Allowed downtime = 43,200 × (1 - 0.999) = 43.2 minutes

Total requests = 100M/month
Allowed failures = 100M × (1 - 0.999) = 100,000 requests

Time elapsed: 10 days (33.3% of month)
Budget used: 15 minutes (34.7% of budget)
Remaining: 28.2 minutes

Burn rate: 34.7% budget / 33.3% time = 1.04× normal
Status: ⚠️ Slightly elevated, monitor closely

# Error Budget Policy — Payment API

## Measurement Window
- SLO: 99.9% over rolling 30 days
- Evaluated: Daily at 00:00 UTC

## Budget Status Tiers

### Tier 1: Healthy (> 50% budget remaining)
- ✅ All development activities allowed
- ✅ Deploy on-demand (no freeze)
- ✅ Experiment with new features

### Tier 2: Warning (20% - 50% budget remaining)
- ⚠️ Increase monitoring frequency
- ⚠️ Freeze risky changes (major refactors)
- ⚠️ Post-mortems mandatory for incidents

### Tier 3: Critical (< 20% budget remaining)
- 🔴 Feature freeze — only reliability fixes
- 🔴 On-call team escalates to senior SRE
- 🔴 Daily review meetings until recovery
- 🔴 No deployments without SRE approval

### Tier 4: Exhausted (0% budget remaining)
- 🚨 Complete freeze — no changes except emergency
- 🚨 CTO-level review required
- 🚨 Root cause analysis + blameless post-mortem
- 🚨 Recovery plan due within 24 hours

## Exceptions
- Security patches: Always allowed (but count toward budget)
- Legal compliance: Always allowed
- Data loss prevention: Always allowed

## Budget Reset
- Automatic: Monthly on 1st day 00:00 UTC
- Manual: CTO approval only (extreme circumstances)

# Naive alert
alert: HighErrorRate
expr: rate(http_errors[5m]) > 0.01

Normal burn rate = 1.0
  (ví dụ: 0.1% errors → burn 0.1% budget/hour)

Burn rate = actual_error_rate / (1 - SLO)

Example:
  SLO = 99.9% → budget = 0.1%/hour normal
  Actual error rate = 1% → burn rate = 1% / 0.1% = 10×

# Prometheus alert rules
groups:
  - name: slo_alerts
    interval: 30s
    rules:
      # Page-worthy: Burn entire budget in 2 hours
      - alert: ErrorBudgetBurnCritical
        expr: |
          (
            sum(rate(http_requests{status=~"5.."}[1h]))
            / sum(rate(http_requests[1h]))
          ) > (14.4 * 0.001)   # 14.4× burn rate for 99.9% SLO
          and
          (
            sum(rate(http_requests{status=~"5.."}[5m]))
            / sum(rate(http_requests[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Critical: Burn entire error budget in 2 hours"
          runbook: "https://runbook.example.com/error-budget-burn"
      
      # Ticket-worthy: Burn entire budget in 6 hours
      - alert: ErrorBudgetBurnHigh
        expr: |
          (
            sum(rate(http_requests{status=~"5.."}[6h]))
            / sum(rate(http_requests[6h]))
          ) > (6 * 0.001)      # 6× burn rate
          and
          (
            sum(rate(http_requests{status=~"5.."}[30m]))
            / sum(rate(http_requests[30m]))
          ) > (6 * 0.001)
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "High: Burn entire error budget in 6 hours"

SLO = 99.9% → budget = 0.1% per 30 days = 43.2 min/month

Critical (14.4×):
  43.2 min / 14.4 = 3 minutes to exhaustion if sustained
  Alert after 2 min → 1 min to respond

High (6×):
  43.2 min / 6 = 7.2 minutes to exhaustion
  Alert after 15 min → still catchable

# Prometheus recording rules
groups:
  - name: sli_rules
    interval: 15s
    rules:
      # Availability SLI
      - record: sli:http_requests:availability:1m
        expr: |
          sum(rate(http_requests{status!~"5.."}[1m]))
          / sum(rate(http_requests[1m]))
      
      # Latency SLI (fraction < 300ms)
      - record: sli:http_requests:latency:1m
        expr: |
          sum(rate(http_request_duration_seconds_bucket{le="0.3"}[1m]))
          / sum(rate(http_request_duration_seconds_count[1m]))
      
      # Error budget remaining (rolling 30d)
      - record: slo:error_budget_remaining
        expr: |
          1 - (
            (1 - sli:http_requests:availability:30d)
            / (1 - 0.999)
          )

┌────────────────────────────────────────────────────┐
│ Error Budget Status                                 │
│                                                     │
│ SLO: 99.9% │ Budget: 43.2 min │ Remaining: 28 min │
│                                                     │
│ ████████████████████████████░░░░░░░░ 65%          │
│                                                     │
│ Burn rate: 1.2× (slightly elevated)                │
│ Time to exhaustion: 23 days (if sustained)         │
│                                                     │
│ [Graph: Budget over time]                          │
│    ▲                                                │
│ 100│╲                                               │
│    │ ╲___                                           │
│  50│     ╲___                                       │
│    │         ╲___                                   │
│   0└─────────────────────────────► Time            │
│     0d        15d        30d                        │
└────────────────────────────────────────────────────┘

# Budget remaining (0-1)
1 - (
  (1 - sli:http_requests:availability:30d{service="payment-api"})
  / (1 - 0.999)
)

# Burn rate (current vs normal)
(
  (1 - sli:http_requests:availability:1h)
  / (1 - 0.999)
) / (1/720)  # 720 hours in 30 days

# Time to exhaustion (days)
(
  slo:error_budget_remaining
  / (1 - sli:http_requests:availability:1h)
) * (1/24)

# ArgoCD ApplicationSet — auto-disable sync khi budget low
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payment-api
spec:
  generators:
    - list:
        elements:
          - env: production
            syncPolicy: |
              {{- if gt .Values.errorBudgetRemaining 0.2 }}
              automated:
                prune: true
                selfHeal: true
              {{- else }}
              # Freeze when budget < 20%
              manual: {}
              {{- end }}

# Pre-deployment check
curl -s "http://prometheus/api/v1/query?query=slo:error_budget_remaining" \
  | jq -r '.data.result[0].value[1]' \
  | awk '{if ($1 < 0.2) exit 1}'  # Exit 1 if < 20%

if [ $? -eq 1 ]; then
  echo "❌ Deployment blocked: Error budget < 20%"
  exit 1
fi

SLO_checkout = SLO_cart × SLO_payment × SLO_inventory
             = 0.999 × 0.999 × 0.999
             = 0.997

→ Target individual SLOs = 99.9% to achieve 99.7% end-to-end

# Define user journey as single SLI
sli:checkout_journey:success:1m =
  (
    requests with:
      - view_product: 200 OK
      - add_to_cart: 200 OK
      - checkout: 200 OK
      - payment: 200 OK
  ) / total_checkout_attempts

Measure: Rolling 30-day window (for accuracy)
Report: Monthly (for business alignment)