Observability

# Rate — requests per second (last 5 min)
rate(http_requests_total[5m])

# Error rate
rate(http_requests_total{status=~"5.."}[5m])
  /
rate(http_requests_total[5m])

# Latency p99
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

# CPU utilization
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory saturation (swap usage)
node_memory_SwapCached_bytes / node_memory_SwapTotal_bytes

# Disk errors
rate(node_disk_io_time_seconds_total[5m])

# Example Prometheus alerting rule
groups:
  - name: myapp.rules
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m])
            /
          rate(http_requests_total[5m]) > 0.05
        for: 2m    # must be true for 2 minutes before firing
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: "Error rate is {{ $value | humanizePercentage }}"
          runbook: "https://wiki.example.com/runbooks/high-error-rate"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "p99 latency above 2 seconds"

Apps expose /metrics endpoint
      │
      ▼
Prometheus scrapes (pulls) metrics every 15s
      │
      ├─ Stores in time-series DB
      ├─ Evaluates alerting rules → Alertmanager → PagerDuty/Slack
      └─ Grafana queries via PromQL

# Instant vector — current value
up                              # 1 = up, 0 = down
http_requests_total             # all series

# Filter by label
http_requests_total{method="GET", status="200"}
http_requests_total{status=~"5.."}    # regex: 5xx errors

# Range vector — values over time window
http_requests_total[5m]         # last 5 minutes of samples

# Functions
rate(http_requests_total[5m])   # per-second rate of increase
irate(http_requests_total[5m])  # instant rate (last 2 samples)
increase(http_requests_total[1h])  # total increase over 1 hour

avg(rate(http_requests_total[5m]))
sum by (status)(rate(http_requests_total[5m]))

# Aggregation over time
avg_over_time(response_time[24h])

# Quantile from histogram
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

// Example panel query (in dashboard JSON)
{
  "expr": "histogram_quantile(0.99, sum by(le) (rate(http_request_duration_seconds_bucket[5m])))",
  "legendFormat": "p99 latency"
}

# Structured logging — always log JSON in production
{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "error",
  "service": "payment-service",
  "trace_id": "abc123",
  "span_id": "def456",
  "user_id": "usr_789",
  "message": "Payment failed",
  "error": "connection timeout",
  "amount": 99.99,
  "currency": "USD"
}

Apps → stdout/stderr
        │
        ▼ (collected by DaemonSet)
Promtail (on each K8s node)
        │
        ▼
Loki (log storage)
        │
        ▼
Grafana (query with LogQL)

# LogQL queries (Grafana / Loki)
{namespace="production", app="myapp"} |= "ERROR"
{app="myapp"} | json | level="error" | line_format "{{.message}}"

User request hits Service A
  → Service A creates a "span" (start time, attributes)
  → Service A calls Service B (propagates trace context via HTTP header)
      → Service B creates a child span
      → Service B queries DB (another child span)
      → Service B responds
  → Service A gets response, closes its span
  → Both services send spans to OTel Collector
      → Collector exports to Jaeger/Tempo/Datadog/etc.

Trace: user request
├── Span: Service A /api/order (100ms total)
│   ├── Span: Service B /checkout (60ms)
│   │   ├── Span: DB query SELECT * FROM... (45ms)  ← bottleneck!
│   │   └── Span: Redis cache check (2ms)
│   └── Span: Send confirmation email (5ms)

// tracing.js — initialize before anything else
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],  // auto-instruments HTTP, Express, pg, redis, etc.
});

sdk.start();

// Manual spans for custom operations
const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('myapp');

async function processPayment(orderId, amount) {
  return tracer.startActiveSpan('process-payment', async (span) => {
    try {
      span.setAttribute('order.id', orderId);
      span.setAttribute('payment.amount', amount);

      const result = await chargeCard(amount);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (error) {
      span.recordException(error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
      throw error;
    } finally {
      span.end();
    }
  });
}

# Scenario: "Payment API is slow for some users"

# Step 1: Metrics alert fires — p99 latency > 2s
# Step 2: Find slow traces in Jaeger/Tempo
#   Filter: service=payment-api, duration > 2s

# Step 3: Open a slow trace — see span waterfall:
Trace (2.3s):
├── POST /api/payments (2.3s)
│   ├── Validate request (1ms)
│   ├── Fetch user (45ms)
│   ├── Check fraud score (1.8s) ← 78% of time here!
│   │   └── HTTP call to fraud-service (1.8s)
│   │       └── DB query in fraud-service (1.7s) ← missing index!
│   └── Process charge (250ms)

# Step 4: Fix — add index to fraud-service DB query
# Step 5: Verify with traces that p99 improved