Observability in Distributed Systems: Logs, Metrics and Tracing

In microservices environments, knowing what is happening inside the application is essential. Observability goes beyond simple monitoring, it allows you to ask your system questions and receive reliable answers.

Three Pillars of Observability

Structured logs, event records in JSON format, facilitating queries.
Metrics, numeric values over time (latency, error rate, CPU usage).
Distributed Tracing, tracking requests that cross multiple services.

Implementation Strategy

Centralize logs using Elastic Stack (Filebeat → Logstash → Elasticsearch → Kibana) or managed services like CloudWatch Logs.
Export metrics to Prometheus and view them in Grafana.
Instrument code with OpenTelemetry, sending spans to Jaeger or Zipkin.

Observability Checklist

Configure JSON logger (e.g.: Winston, Bunyan).
Set service.name and environment in all logs.
Export standard metrics (process_cpu_seconds_total, process_resident_memory_bytes).
Instrument HTTP clients and servers with OpenTelemetry.
Create latency, error rate and throughput dashboards.
Define SLA alerts (e.g. latency > 500ms, error > 1%).
Document ID correlation (trace_id, span_id).

JSON Logger Example (Node.js)

const winston = require('winston');

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json()
  ),
  defaultMeta: {
    service: process.env.SERVICE_NAME,
    environment: process.env.NODE_ENV
  },
  transports: [new winston.transports.Console()]
});

module.exports = logger;

Custom Metric Example with Prometheus (Node.js)

const client = require('prom-client');

const requestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duração das requisições HTTP em segundos',
  labelNames: ['method', 'route', 'status_code']
});

app.use((req, res, next) => {
  const end = requestDuration.startTimer({ method: req.method, route: req.path });
  res.on('finish', () => end({ status_code: res.statusCode }));
  next();
});

Tracing Example with OpenTelemetry (Node.js)

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');

const provider = new NodeTracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(new JaegerExporter({
  endpoint: process.env.JAEGER_ENDPOINT
})));
provider.register();

Conclusion

Implementing observability requires discipline, but brings immediate returns: early detection of failures, reduction in MTTR (Mean Time To Recovery) and confidence to scale critical systems.

How do you monitor your services? Share your favorite tools!