Every exchange API fails in the same ways: REST calls time out, rate limits bite, and the streaming connection — which you might hold open for hours — drops, usually at the worst possible moment. A trading client that treats these as exceptional is a client that loses money during an outage; a resilient client treats them as normal events with defined responses. This post is the resilience design for an exchange client in Java, split across the two failure surfaces every exchange exposes: request/response REST and long-lived streaming.
The two failure surfaces
An exchange client is really two clients:
- REST — short calls: place order, cancel order, get account. Failures are per-call: timeout, 5xx, rate-limit 429. Retry and circuit-breaking apply.
- Streaming — one long-lived connection carrying a continuous feed of market and order updates. Failure is binary: the connection is up or it isn’t. When it drops, everything subscribed is lost until you reconnect and resubscribe.
Mixing the two resilience strategies up — circuit-breaking a streaming connection, or retrying a stream reconnect like a REST call — is where client designs come unstuck.
REST: retries with exponential backoff and jitter
A timeout or a 429 is frequently transient. The default response is to retry — but naive retries (immediate, or fixed-interval) turn a blip into a self-inflicted load spike, and synchronized retry storms from many clients are how APIs fall over completely.
@Retryable(
retryFor = { TimeoutException.class, RateLimitException.class },
maxAttempts = 4,
backoff = @Backoff(delay = 200, multiplier = 2.0, maxDelay = 5000))
public OrderResponse placeOrder(PlaceOrderRequest request) { ... }
Exponential backoff with a jitter term — and a hard cap — is the standard shape. The crucial rule for trading: only retry idempotent or safely-replayable calls. Re-sending a place-order that already succeeded creates a duplicate. If the API does not give you a client-generated order ID to dedupe on, a retried order placement is a risk decision, not a mechanical retry. The retry annotations in Spring post covers the configuration mechanics.
REST: the circuit breaker on the order path
Retries handle transient failures; they make sustained failures worse — every retry burns a request into an API that is already down. The circuit breaker stops the bleeding: after N consecutive failures the circuit opens and calls fail fast without hitting the API, giving it time to recover, then half-opens to probe.
@CircuitBreaker(name = "orderApi", fallbackMethod = "orderFallback")
public OrderResponse placeOrder(PlaceOrderRequest request) { ... }
The order path is the right place for a breaker because it is the path where failing fast beats queuing: a breaker that opens during an exchange outage lets you surface “trading is suspended” to the operator instead of piling up failed orders. The /circuit-breaker demo on this site animates the CLOSED → OPEN → HALF-OPEN cycle with configurable thresholds — the same state machine the client runs, minus the animation. For the Resilience4j configuration in depth, the Resilience4j circuit breaker post has it.
Streaming: heartbeat and timeout
The streaming connection’s health signal is the heartbeat: the exchange sends a heartbeat message on a fixed interval, and a client that stops receiving anything — data or heartbeat — knows the connection is dead long before a socket error surfaces. The client tracks the time since the last message and treats a silent gap (say, 2.5 heartbeats) as a connection failure:
ScheduledExecutorService watchdog = Executors.newSingleThreadScheduledExecutor();
watchdog.scheduleAtFixedRate(() -> {
if (now() - lastMessageAt > HEARTBEAT_TIMEOUT) {
reconnect(); // close, back off, reconnect, resubscribe
}
}, 1, 1, TimeUnit.SECONDS);
Heartbeats are also the rate-limit signal on the stream: sending market data when you haven’t subscribed is answered with a heartbeat, and a client that treats heartbeats as noise misses the exchange telling it the subscription is invalid. The streaming API subscriptions post covers the connection and subscription mechanics; this is the layer that keeps it alive.
Streaming: reconnect with backoff and resubscribe
Reconnecting is not the same as recovering. After the TCP connection comes back, every market and order the client was subscribed to is gone — the exchange does not remember. The recovery sequence is:
- Back off — the exchange is often still flapping; immediate reconnect loops are the signature of a broken client. Exponential backoff with a cap, resetting on a connection that survives a full heartbeat cycle.
- Reconnect and re-authenticate — the session token is frequently invalidated by the drop; re-auth before resubscribe.
- Resubscribe to everything — replay the subscription list from the client’s own state, not from memory of what was sent.
- Handle the gap — the stream resumes with new updates, not history. Anything that happened while disconnected is missed. For market prices that is usually acceptable (the next image is fresh); for order updates it is not — which is why the next step matters.
Replaying missed state
The honest name for a streaming outage is lost updates. The resilient client does not pretend otherwise: after reconnect it fetches full state for everything it tracks — current market image, open orders, account balance — from the REST API, and reconciles. For an order-management client, “the stream dropped” must trigger “what actually happened while I was blind?” — an idempotent reconciliation against the exchange’s own order list — or the client’s local view silently diverges from the exchange’s truth. This is the step most tutorial-level streaming clients omit, and the one that separates a demo from a trading system.
The client lifecycle, assembled
The whole design is one loop with defined states:
CONNECTED
→ (heartbeat timeout) → RECONNECTING
RECONNECTING
→ (backoff, re-auth) → RESUBSCRIBING
RESUBSCRIBING
→ (subscriptions replayed) → RECONCILING
RECONCILING
→ (full state fetched) → CONNECTED
Every state transition is logged with its reason, every retry has a cap, and every failure that outlives the retry budget escalates to the operator instead of looping silently. That last rule is the discipline of the whole design: resilience is not “never fail” — it is “fail according to a plan, and tell someone when the plan runs out”.
If you’re building a client against an exchange API and want the reconnect and recovery loop designed properly, get in touch.
