3-pillar observability strategy
Logs, metrics, and traces — wired together so you never fly blind in production.
Structured logging
Every log line carries a trace ID, a severity level, and a canonical event name. Ship JSON to stdout — let your collector handle routing. No printf debugging in production paths.
- • Correlation IDs on inbound requests
- • Redact PII at the logger boundary
- • Sample at DEBUG, keep INFO and above always-on
Real-time metrics
Expose RED metrics — Rate, Errors, Duration — per endpoint. Push to a time-series store with 10-second granularity. Alert on error budget burn, not raw spike counts.
- • p50 / p95 / p99 latency histograms
- • Saturation gauges for thread pools and queues
- • Circuit-breaker state as a first-class metric
Distributed tracing
Propagate W3C trace context across every service boundary. Annotate spans with business-relevant attributes — user ID, tenant, feature flag state. A trace without context is just a pretty waterfall.
- • Span events for cache hits, retries, and fallbacks
- • Tail-based sampling for error and high-latency traces
- • Link traces to log entries via trace ID injection
Wiring it together
The three pillars collapse into one when your trace ID appears in every log line and your metrics dashboard links directly to exemplar traces. Start with structured logs, add metrics next sprint, and layer tracing once you have more than two services.