Chapitre 14 : SRE (Site Reliability Engineering)

14.1 Introduction au SRE

14.1.1 Qu'est-ce que le SRE ?

Le Site Reliability Engineering est une discipline qui applique les principes du genie logiciel aux operations d'infrastructure. Cree par Google en 2003, le SRE se positionne entre le developpement et les operations.

Definition de Google :

"SRE is what happens when you ask a software engineer to design an operations team."

14.1.2 Les principes fondamentaux

1. Ops is a software engineering problem : Automatiser tout ce qui peut l'etre
2. Hiring for SRE : 50-50 mix de software engineers et sysadmins
3. Risk acceptance : 100% de disponibilite n'est pas le bon objectif
4. Error budgets : 100% de fiabilite empeche l'innovation
5. Monitoring : Les alertes doivent etre actionnables
6. Toil reduction : Max 50% de temps consacre aux operations
7. Incident management : Processus rigoureux et blameless postmortems

14.1.3 Les 4 niveaux de monitoring SRE

Level 1: Alerts (alerte immediate, action requise)
Level 2: Tickets (action necessaire mais pas immediate)
Level 3: Logs (information, pas d'action)
Level 4: Metrics (analysis, tendances)

14.2 SLI, SLO, SLA

14.2.1 Definitions

14.2.2 Choisir ses SLIs

SLIs recommandes par le SRE Book :

Exemple de definition :

# sli-definition.yaml
slis:
  availability:
    name: "API Availability"
    measurement: >
      ratio of successful HTTP requests (status 200-499)
      to total requests over 1 minute windows
    data_source: prometheus
    query: |
      sum(rate(http_requests_total{status!~"5.."}[1m])) /
      sum(rate(http_requests_total[1m]))

  latency:
    name: "API Latency p99"
    measurement: "99th percentile of response time"
    data_source: prometheus
    query: |
      histogram_quantile(0.99,
        sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

14.2.3 Definir ses SLOs

# slo-definition.yaml
slos:
  - name: "API Availability SLO"
    target: 99.9%
    window: 30d
    description: "API availability over rolling 30 days"
    sli: availability

  - name: "API Latency SLO"
    target: 99.0% (p99 < 500ms)
    window: 7d
    description: "p99 latency under 500ms over 7 days"
    sli: latency

  - name: "API Error Rate SLO"
    target: 99.95%
    window: 7d
    description: "Error rate < 0.05%"
    sli: error_rate

14.2.4 Multi-Window, Multi-Burn-Rate Alerts

# slo-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: slo-alerts
spec:
  groups:
    - name: slo-rules
      rules:
        - alert: SLOWarning
          expr: |
            (
              1 - (sum(rate(http_requests_total{status!~"5.."}[1h]))
                   / sum(rate(http_requests_total[1h])))
            ) > 0.001  # < 99.9% over 1 hour
          for: 6h
          labels:
            severity: warning
          annotations:
            summary: "SLO burn rate is high"

        - alert: SLOCritical
          expr: |
            (
              1 - (sum(rate(http_requests_total{status!~"5.."}[5m]))
                   / sum(rate(http_requests_total[5m])))
            ) > 0.01  # < 99% over 5 minutes
          for: 30m
          labels:
            severity: critical
          annotations:
            summary: "SLO burn rate critical"

14.3 Error Budget

14.3.1 Concept

L'error budget est la quantite d'erreur acceptable sur une periode :

Error Budget = 100% - SLO

Exemple: SLO = 99.9%
Error Budget = 0.1% de 30 jours = 43.2 minutes d'indisponibilite

14.3.2 Utilisation de l'Error Budget

# error_budget.py
from datetime import datetime, timedelta

class ErrorBudget:
    def __init__(self, slo_percentage, window_days=30):
        self.slo = slo_percentage / 100
        self.window = timedelta(days=window_days)
        self.total_budget = self.window.total_seconds() * (1 - self.slo)
        self.consumed = 0
    
    def record_failure(self, duration_seconds):
        self.consumed += duration_seconds
    
    @property
    def remaining(self):
        return self.total_budget - self.consumed
    
    @property
    def burn_rate(self):
        """Taux de consommation de l'error budget"""
        return self.consumed / self.total_budget
    
    def should_deploy(self):
        """Devrait-on autoriser les deploiements ?"""
        return self.consumed < self.total_budget * 0.75  # 75% consomme = freeze

Regles de decision basees sur l'error budget :

error_budget_policy:
  thresholds:
    - name: "green"
      consumed: 0-50%
      action: "Deploiements autorises"
    - name: "yellow"
      consumed: 50-75%
      action: "Deploiements ralentis, revue approbation"
    - name: "red"
      consumed: 75-100%
      action: "FREEZE: deploiements bloques, priorite stabilite"
    - name: "exhausted"
      consumed: 100%+
      action: "Incident critique, rollback obligatoire"

14.4 Toil Reduction

14.4.1 Qu'est-ce que la toil ?

La toil est un travail manuel, repetitif, automatisable, sans valeur durable :

Exemples de toil :

• Reinitialiser manuellement des mots de passe
• Repondre a des alertes non-actionnables
• Deployer manuellement des applications
• Mettre a jour des configurations a la main
• Reboot des serveurs un par un

Exemples de NON-toil :

• Debug d'un incident complexe
• Mise en place d'une nouvelle automatisation
• Design d'une architecture
• Code review

14.4.2 Strategie de reduction

# toil-reduction-plan.yaml
toil_reduction:
  target: "Moins de 50% de temps en toil"
  
  strategies:
    - name: "Automatisation"
      actions:
        - "Identifier les taches repetitives"
        - "Creer des scripts/outils"
        - "Deployer des self-service portals"
    
    - name: "Elimination"
      actions:
        - "Arreter les processus sans valeur"
        - "Supprimer les alertes non-actionnables"
        - "Deprecated des anciens systemes"
    
    - name: "Delegation"
      actions:
        - "Former les equipes produit"
        - "Creer des runbooks"
        - "Self-service pour les operations courantes"

14.5 Incident Management

14.5.1 Niveaux de severite

# severity-levels.yaml
severity_levels:
  SEV1:
    name: "Critical"
    description: "Service completement indisponible"
    response_time: "15 minutes"
    examples:
      - "Tous les utilisateurs impactes"
      - "Perte de donnees"
      - "Violation de securite"
    
  SEV2:
    name: "High"
    description: "Fonctionnalite majeure degradee"
    response_time: "30 minutes"
    examples:
      - "Performance degradee pour tous"
      - "Fonctionnalite critique indisponible"
    
  SEV3:
    name: "Medium"
    description: "Impact mineur, pas d'utilisateurs impactes"
    response_time: "4 hours"
    examples:
      - "Erreur sur une fonctionnalite mineure"
      - "Degradation des performances"
    
  SEV4:
    name: "Low"
    description: "Pas d'impact utilisateur"
    response_time: "8 hours"
    examples:
      - "Bug cosmétique"
      - "Tache operationnelle"

14.5.2 Processus de gestion d'incident

1. DETECTION
   │
2. TRIAGE (severite, escalation)
   │
3. RESPONSE (mitigation, communication)
   │
4. RESOLUTION (fix permanent)
   │
5. POSTMORTEM (blameless, action items)
   │
6. FOLLOW-UP (implementation des actions)

Exemple de postmortem :

# Postmortem: Incident INC-2024-001

## Date: 2024-01-15 14:30 UTC
## Severite: SEV1 (Critical)
## Duree: 47 minutes
## Impact: 100% des utilisateurs impactes, 15K requetes perdues

## Chronologie
- 14:30 - Alerte: Error rate > 10% (pager)
- 14:32 - Engineer 1 respond
- 14:35 - Diagnostic: Database connection pool epuise
- 14:40 - Decision: Restart database pod
- 14:45 - Database redemarre
- 14:47 - Trafic retabli
- 15:02 - Declaration de fin d'incident
- 15:17 - Postmortem debute

## Causes racines
1. **Cause directe**: Connection pool epuise suite a un pic de trafic
2. **Cause sous-jacente**: Pas de max_connections configure
3. **Cause systemique**: Pas de test de charge avant deploiement

## Actions
| Action | Owner | Due Date |
|--------|-------|----------|
| Configurer max_connections | DB Team | 2024-01-16 |
| Ajouter test de charge au pipeline | DevOps | 2024-01-30 |
| Creer runbook pour restart DB | SRE | 2024-01-20 |

## Lessons learned
- L'alerte error rate a bien fonctionne
- Le runbook etait incomplet
- Le max_connections doit etre revu regulierement

14.6 Capacity Planning

14.6.1 Approche Data-Driven

# capacity_planning.py
import numpy as np
from datetime import datetime, timedelta

class CapacityPlanner:
    def __init__(self, metrics_history_days=90):
        self.history_days = metrics_history_days
    
    def predict_growth(self, metric_values):
        """Prediction lineaire simple de la croissance"""
        x = np.arange(len(metric_values))
        coeffs = np.polyfit(x, metric_values, 1)
        return coeffs[0]  # pente
    
    def threshold_date(self, current_usage, growth_rate, threshold=0.8):
        """Date a laquelle on atteindra un seuil d'utilisation"""
        if growth_rate <= 0:
            return None
        
        capacity = current_usage / threshold
        days_remaining = (capacity - current_usage) / growth_rate
        return datetime.now() + timedelta(days=days_remaining)
    
    def scaling_recommendation(self, current_usage, growth_rate):
        days_to_80pct = self.threshold_date(current_usage, growth_rate)
        
        if days_to_80pct and days_to_80pct.days < 30:
            return {
                "action": "URGENT: Scale immediately",
                "days_remaining": days_to_80pct.days
            }
        elif days_to_80pct and days_to_80pct.days < 90:
            return {
                "action": "WARNING: Plan scaling",
                "days_remaining": days_to_80pct.days
            }
        else:
            return {"action": "OK: No action needed"}

14.6.2 Signaux de capacity planning

14.7 Chaos Engineering

14.7.1 Principes du Chaos Engineering

"Chaos Engineering is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions in production." - Principles of Chaos Engineering

Les 5 principes :

1. Definir l'etat stable du systeme
2. Emettre des hypotheses sur cet etat stable
3. Introduire des variables chaotiques (pannes, latence)
4. Tenter de refuter les hypotheses
5. Automatiser les experiences

14.7.2 Chaos Mesh

# chaos-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-kill-experiment
  namespace: chaos-engineering
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: api
  duration: "60s"
  scheduler:
    cron: "@every 6h"
---
apiVersion: chaos-mesh.org/v1alpha1
kind: HTTPChaos
metadata:
  name: http-delay-experiment
spec:
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      app: api
  target: Request
  port: 8080
  delay:
    latency: "2000ms"
    correlation: "50"
    jitter: "100ms"
  duration: "5m"
  scheduler:
    cron: "@every 24h"
---
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: network-partition-experiment
spec:
  action: partition
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      app: api
  direction: both
  target:
    mode: all
    selector:
      namespaces:
        - production
      labelSelectors:
        app: database
  duration: "30s"

14.8 DORA Metrics

14.8.1 Les 4 metrics DORA

Les DORA (DevOps Research and Assessment) metrics sont les 4 indicateurs cles de la performance DevOps :

14.8.2 Mesure des DORA metrics

# Exemple de mesure avec GitHub API
#!/bin/bash

# Deployment Frequency
gh api repos/YOUR_ORG/YOUR_REPO/deployments --jq 'length'

# Lead Time (du commit au deploiement)
gh api repos/YOUR_ORG/YOUR_REPO/pulls --jq \
  '[.[] | {number, created_at, merged_at}]'

# MTTR (Mean Time To Recover)
# Temps entre le debut de l'incident et le fix
gh issue list --label incident --json createdAt,closedAt

# Change Failure Rate
# Nombre de deploiements avec incident / deploiements total

14.9 Production Readiness Review

14.9.1 Checklist PRR

# production-readiness-review.yaml
services:
  - name: "myapp"
    version: "2.0.0"
    owner: "Team Alpha"
    
    reliability:
      - slo_defined: true
      - error_budget_defined: true
      - dependencies_mapped: true
      - single_points_of_failure_identified: true 
    
    observability:
      - metrics_implemented: true
      - dashboards_created: true
      - alerts_configured: true
      - logs_centralized: true
      - tracing_enabled: true
    
    security:
      - vulnerability_scan: true
      - secrets_management: true
      - network_policies: true
      - image_signing: true
    
    operations:
      - runbook_created: true
      - on_call_rotation: true
      - backup_strategy: true
      - disaster_recovery_plan: true
    
    performance:
      - load_testing_done: true
      - capacity_plan: true
      - autoscaling_configured: true
      - resource_limits_set: true

Resume

• SRE applique le genie logiciel aux operations
• SLI mesure, SLO fixe l'objectif, SLA est l'engagement contractuel
• L'error budget permet d'equilibrer fiabilite et innovation
• La toil doit etre automatisee (< 50% du temps SRE)
• Les incidents suivent un processus rigoureux (detection → postmortem)
• Le capacity planning est guide par les donnees
• Le chaos engineering valide la resilience du systeme
• Les DORA metrics mesurent la performance DevOps
• Le Production Readiness Review assure la qualite avant mise en production