Collecting every metric you can think of produces dashboards, not observability. A dashboard with sixty panels answers nothing; an on-call engineer staring at a wall of graphs cannot tell whether the service is healthy. The cure is a small, opinionated metric set — and the two naming frameworks that produce one are RED for request-driven services and USE for resources. This post explains both, how to record them with Micrometer, and — the part most posts skip — which ones deserve an alert.
RED for request-driven services
RED is the method for anything that serves requests: Rate, Errors, Duration. Three numbers answer “is this service working?”:
- Rate — requests per second. Is traffic what we expect?
- Errors — requests failing (5xx, timeouts). Are we failing?
- Duration — latency distribution. Are we slow?
These are the first three panels on every dashboard because they are the three questions anyone asks first. Everything else — JVM memory, thread counts, GC pauses — is a debugging detail you drill into after RED points at a problem.
USE for resources
USE is the resource-oriented counterpart: Utilisation, Saturation, Errors. It applies to anything with a capacity: thread pools, database connection pools, CPU, disk.
- Utilisation — what fraction of capacity is in use?
- Saturation — how much work is queued beyond the capacity?
- Errors — how many resource operations fail?
Saturation is the one people forget, and it is the most predictive. A Tomcat thread pool at 60% utilisation with a growing queue depth is failing soon; utilisation alone will not tell you that. For a Spring Boot service, the resources that matter are the HTTP thread pool, the HikariCP pool, and the Kafka consumer threads.
Percentiles, not averages
Averages hide the problem. If 99 requests take 10ms and one takes 2 seconds, the average is 30ms and the p99 is 2s — and the p99 is the number your users feel. Record latency as a distribution, not a mean:
Timer.Sample sample = Timer.start(registry);
// ... the request ...
sample.stop(registry.timer("http.server.requests",
"uri", uri, "method", method));
Micrometer’s Timer records a histogram under the hood, and Prometheus’s histogram_quantile derives p50, p95 and p99 from it. If you are already collecting metrics with Micrometer and Prometheus — the observability with Actuator, Micrometer and Prometheus post covers the plumbing — switching from averages to percentiles is purely a query-side change: histogram_quantile(0.99, rate(...)) instead of avg(...).
Recording the set with Micrometer
Micrometer gives you four primitives, and each maps to one of the RED/USE roles:
- Counter — monotonically increasing events: request count, error count. RED’s Rate and Errors.
- Timer — latency distributions. RED’s Duration.
- Gauge — current state that goes up and down: pool utilisation, queue depth. USE’s Utilisation and Saturation.
- DistributionSummary — arbitrary sizes: batch sizes, payload sizes.
The custom metrics with Micrometer post covers each primitive in depth; the point here is that you only need a handful. For a typical Spring Boot service the full RED/USE set is about ten metrics.
What to alert on
This is where most setups go wrong: everything becomes an alert, so nothing is. The discipline is simple — alert on RED, not on resource trivia, and express alerts as SLO burn, not static thresholds:
- Errors — error ratio above a threshold for a sustained window. “More than 1% of requests failed for 5 minutes” is an alert. “One 500 occurred” is a log line.
- Latency — p99 above budget for a sustained window, not a single slow spike.
- Saturation — queue depth or pool saturation above a level that predicts failure, not utilisation crosses.
- SLO burn — the modern replacement for all three: track a rolling error budget, and page when the budget is burning faster than the SLO allows, at different severities for fast vs slow burn.
Static thresholds on individual metrics produce pages that say “p99 is 600ms” with no context on whether that threatens a promise. Burn-rate alerts answer the real question: are we on track to keep this month’s SLO? That is a page-worthy question.
The starter set
A production Spring Boot service can run on this, and no more:
| Metric | Type | Alert |
|---|---|---|
| Request rate | Counter/Rate | no — traffic dip is a separate story |
| Error ratio | Counter derived | yes — SLO burn |
| p50/p95/p99 latency | Timer histogram | yes — SLO burn |
| HTTP thread pool utilisation + queue | Gauge | yes — saturation only |
| HikariCP pool utilisation + wait time | Gauge/Timer | yes — saturation only |
| Kafka consumer lag | Gauge | yes — lag beyond budget |
Ten metrics, four alerts. Everything else — GC, heap, file descriptors — is diagnosis material, recorded for when you drill in, not for paging anyone.
The habit to build: every new metric earns its place by answering one of the RED/USE questions. If it doesn’t, it is noise — and noise is what makes real alerts invisible.
If you’re standing up observability for a Spring Boot service and want a metric set that tells you what’s actually wrong, start a conversation.
