I’ve applied both across long-lived, high-stakes systems — from real-time financial data pipelines to government benefit platforms — and here’s how I think about the choice.
Both architectures aim to:
Where they differ is in structure and emphasis.
Hexagonal Architecture (Ports and Adapters) focuses on interaction points. The domain sits at the centre, and everything outside it — HTTP, Kafka, persistence, third-party APIs — is an adapter connected through a port. The model is interaction-driven and maps naturally to event-driven systems.
Strengths:
Where I use it: At DWP Digital and Mosaic Smart Data, Hexagonal Architecture was the natural choice. Multiple entry points (Kafka events, HTTP, scheduled jobs) and the need for domain logic untouched by infrastructure made Ports and Adapters the right foundation.
Clean Architecture introduces concentric dependency layers with strict inward-pointing rules. It emphasises use cases as first-class citizens alongside entities, and provides more prescriptive guidance on layer responsibilities.
Strengths:
Where I use it: When domain complexity grows and teams expand, I selectively introduce Clean Architecture concepts — particularly explicit use case objects — to bring additional structure where Hexagonal Architecture alone starts to feel ambiguous.
In Spring Boot systems, I start with Hexagonal Architecture. It gives flexibility without unnecessary ceremony, and the port/adapter model maps directly to the Spring component model. When systems grow large, or when teams expand to the point where boundary ambiguity causes friction, I layer in Clean Architecture concepts where they add genuine clarity — not as a rewrite, but as a targeted evolution.
The cleanest systems I’ve worked on borrowed from both. Hexagonal Architecture as a foundation, with explicit use case objects from Clean Architecture where domain complexity demanded them. The goal is always the same: protect your domain from the churn of frameworks and infrastructure, and keep business logic testable without spinning up a database or a Kafka broker.
You don’t need to pick sides. Pick what solves your actual problem.
]]>Let’s clear that up.
One of the most persistent myths is that object-oriented programming is fundamentally about managing mutable state. It isn’t.
Objects are not data structures. Objects may use data structures internally, but those details are deliberately hidden. This is why fields are private. From the outside, you cannot see state at all—you can only see behaviour.
An object is best understood as a bag of functions, not a bag of data.
When objects are treated primarily as data holders—especially when exposed directly via getters and setters—that’s not good OO, it’s a design smell. This confusion is why tools like ORMs muddy the waters. They don’t map relational data to objects; they map relational data to data structures that merely look like objects.
Good OO keeps data hidden and exposes intention through behaviour.
Functional Programming doesn’t escape this reality either. Every functional program ever written is composed of functions that operate on data. So is every OO program.
It’s common to hear OO described as data and functions bound together, but that’s true of all programs. The real difference is not that functions and data are bound—it’s how much discipline is imposed on changing them.
At that level, OO and FP are not opposites at all.
The meaningful differences between OO and FP come down to what each style restricts.
A true functional language does not allow assignment. You cannot change the value of a variable once it is defined. Mutation is either impossible or heavily constrained behind explicit ceremony.
This has profound consequences:
Most functional languages do allow mutation eventually—but only after you’ve jumped through deliberate hoops. And because of that friction, most code simply doesn’t mutate state at all.
That’s not an accident. That’s the point.
OO’s real contribution is polymorphism.
Polymorphism is implemented with function pointers. In low-level languages, managing those pointers manually is painful and error-prone. Object-oriented languages do this work for you. They initialise them, marshal them, and invoke them safely.
In languages like Java, every method call is polymorphic. Every call is an indirect call through a function reference.
This is what OO really gives you: safe, ubiquitous polymorphism without manual pointer management.
FP restricts assignment. OO restricts how functions are referenced and invoked. These concerns are orthogonal.
There is nothing preventing a system from imposing both disciplines. You can write object-oriented functional code, and when done well, it’s extremely powerful.
Polymorphism inverts source-code dependencies.
At runtime, a caller still depends on the callee. But at compile time, both depend on an abstraction. This inversion allows implementations to be swapped without touching core business logic.
This is the foundation of plugin architectures, hexagonal architecture, and dependency inversion.
The benefit of restricting assignment is simple: you cannot corrupt shared state if you never mutate it.
Functional-style immutability dramatically reduces race conditions, concurrency bugs, and temporal coupling — critical properties in modern multi-core systems.
Choosing FP does not make design principles optional. Choosing OO does not make them automatic.
Good design is independent of paradigm.
OO is good when you understand what it actually is. FP is good when you understand what it actually is. Combining them thoughtfully is often better than treating either as dogma.
If you’re still arguing OO vs FP, you’re arguing the wrong thing.
]]>That experience is why Hexagonal Architecture (also known as Ports and Adapters) has become my default approach for building systems that stay maintainable and continue to evolve without fear.
Hexagonal Architecture is a design approach that places your domain logic at the centre of the system and treats everything else—databases, messaging systems, HTTP APIs, frameworks—as replaceable details.
At a high level:
The key idea is simple but powerful:
Your business logic should not depend on infrastructure.
Infrastructure should depend on your business logic.
This inversion is what keeps systems flexible under real-world pressure.
On modern platforms—especially event-driven ones—you rarely have a single interface. A service might be triggered by HTTP requests, Kafka events, scheduled jobs, or batch reprocessing. Without a clear boundary, logic gets duplicated or embedded in the wrong places. Hexagonal Architecture gives you one place where business decisions live, regardless of how the system is driven.
Ports are interfaces owned by the domain:
public interface ClaimRepository {
Optional<Claim> findById(ClaimId id);
void save(Claim claim);
}
Adapters translate infrastructure concerns into domain interactions, keeping frameworks out of the core.
Because the domain depends only on ports, business logic can be tested without Spring, Kafka, or databases. This leads to faster tests, clearer intent, and far greater confidence when making changes.
Hexagonal Architecture isn’t about purity — it’s about protecting what matters most: your business logic. If you’re building systems that must evolve safely over time, it’s one of the most effective architectural patterns you can adopt.
]]>@Retryable annotation.
I’ve used Spring Retry in real-time data pipelines at Mosaic Smart Data, where transient failures from external financial institution APIs were a fact of life. In this post I’ll walk through how I applied it, including exponential backoff and selective exception handling.
When a method calls an external API, that call can fail due to reasons beyond our control — temporary outages, throttling, or network latency spikes. In such scenarios, retrying the operation after a delay is often enough to recover gracefully.
Without a retry mechanism, these transient errors can lead to unnecessary failures, noisy logs, and degraded user experience. Implementing retry logic manually can also clutter your code and lead to subtle bugs.
Spring Retry provides a declarative, configurable, and centralized way to handle this.
To use Spring Retry, you need to add the dependency and enable it:
<!-- pom.xml -->
<dependency>
<groupId>org.springframework.retry</groupId>
<artifactId>spring-retry</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
Then, enable retry support in your Spring Boot application:
@SpringBootApplication
@EnableRetry
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
The @Retryable annotation can be placed on any Spring-managed bean method to make it automatically retry on failure.
Here’s the basic structure:
@Retryable(
maxAttempts = 5,
backoff = @Backoff(delay = 1000, multiplier = 2.0)
)
public ApiResponse callExternalApi(String param) {
// potentially flaky operation
}
maxAttempts specifies how many times to try before giving up.@Backoff controls the delay between attempts.
delay = 1000 means start with a 1-second delay.multiplier = 2.0 means each subsequent retry doubles the delay (exponential backoff).If the method keeps failing, the last thrown exception is propagated.
One of the strengths of Spring Retry is that its parameters can be externalized to configuration, allowing different environments (e.g. dev vs prod) to tune retry behavior without code changes.
@Retryable(
maxAttemptsExpression = "${retry.max.attempts:5}",
noRetryFor = {ApiException.class, DailyLimitExceededException.class},
backoff = @Backoff(
delayExpression = "${retry.backoff.delay:1000}",
multiplierExpression = "${retry.backoff.multiplier:2.0}"
)
)
public ApiResponse callExternalApi(String param) {
...
}
maxAttemptsExpression lets you read values from application properties, with 5 as a fallback default.noRetryFor specifies exceptions that should not trigger retries — for example, business logic errors likeDailyLimitExceededException should fail immediately.delayExpression and multiplierExpression allow runtime configuration of backoff behavior.Sample application.yml
retry:
max:
attempts: 5
backoff:
delay: 1000
multiplier: 2.0
In my application, I have an abstract base class called ProviderService, which provides common functionality for
multiple data provider integrations. One of its key methods, lookup(String lookup), queries an external API.
Here’s how the retry mechanism was applied:
@LogExecutionTime
@Retryable(
maxAttemptsExpression = "${retry.max.attempts:5}",
noRetryFor = {ApiException.class, DailyLimitExceededException.class},
backoff = @Backoff(
delayExpression = "${retry.backoff.delay:1000}",
multiplierExpression = "${retry.backoff.multiplier:2.0}"
)
)
public abstract E lookup(final String lookup)
throws ApiException, DailyLimitExceededException;
This allows every concrete provider (e.g. CRM, Zone, Prem) to inherit the retry behaviour automatically, without repeating logic in each subclass.
If a lookup fails due to a transient error (e.g. timeout), Spring will automatically retry it up to 5 times with exponential backoff. If the failure is due to a business rule (e.g. daily limit exceeded), the method will fail fast without retries.
Spring Retry is a powerful tool for adding resilience and fault tolerance to your application with minimal code.
By using @Retryable:
This small annotation can have a significant impact on the stability and reliability of your integration-heavy services.
]]>Every garbage collector must periodically pause application threads to collect unreachable objects. For most applications, a 50ms pause every 30 seconds is irrelevant. For a system that evaluates trading signals from a live streaming feed, a 50ms pause is a missed market event. For an in-play trading system, it’s a potentially unhedged position.
Understanding the trade-off: throughput vs pause time. G1GC optimises for throughput with bounded pauses. ZGC optimises for minimal pause time. Serial/Parallel GC maximise throughput with no pause-time guarantees. For low-latency systems, ZGC is almost always the right choice on Java 21.
G1GC is the default collector since Java 9. It divides the heap into regions and prioritises collection of regions with the most garbage (“garbage-first”). Key configuration:
-XX:+UseG1GC # explicit (default on Java 9+)
-XX:MaxGCPauseMillis=100 # target max pause (not guaranteed)
-XX:G1NewSizePercent=20 # young generation minimum
-XX:G1MaxNewSizePercent=40 # young generation maximum
-XX:G1HeapRegionSize=8m # region size (power of 2, 1m-32m)
-XX:InitiatingHeapOccupancyPercent=35 # start concurrent marking at 35% heap
G1GC’s MaxGCPauseMillis is a target, not a hard limit. Under allocation pressure it may be breached. For my Betfair streaming data processing, G1GC worked well for pre-race analysis (latency requirements ~100ms) but was too unpredictable for in-play.
ZGC (available since Java 15, production-ready Java 21) is a concurrent collector designed for sub-millisecond pause times regardless of heap size. Most GC work happens concurrently with application threads:
-XX:+UseZGC
-XX:SoftMaxHeapSize=4g # soft upper bound — ZGC may exceed in GC pressure
-Xmx6g # hard upper bound
-XX:ZCollectionInterval=0 # let ZGC choose collection frequency
-XX:ZUncommitDelay=300 # return memory to OS after 300s of inactivity
ZGC’s stop-the-world pauses are limited to root scanning — typically under 1ms, often under 500μs on modern hardware. For a Betfair in-play trading system, this is transformative. The trading loop continues running while GC collects concurrently.
The trade-off: ZGC uses more CPU for concurrent work. If your system is CPU-bound, ZGC’s background threads compete with application work. For IO-bound workloads (most trading systems), this is rarely a problem.
Don’t tune without measuring first. Enable GC logging:
-Xlog:gc*:file=/var/log/app/gc.log:time,uptime,level,tags:filecount=5,filesize=20m
This writes structured GC logs you can analyse with tools like GCViewer or GCEasy. Look for:
GC(0) Pause Young (Normal) 150ms — too high for low-latency)GC(5) Pause Full (Ergonomics) — should be rare)JDK Flight Recorder is even more powerful:
-XX:+FlightRecorder
-XX:StartFlightRecording=duration=60s,filename=/tmp/recording.jfr,settings=profile
In JDK Mission Control, the GC events view shows pause distribution, allocation rates, and top allocation sites. The allocation profiler identifies which code paths allocate most — the first step in reducing GC pressure.
The best GC optimisation is allocating less. Common patterns in high-throughput Java systems:
Object pooling for frequently created/discarded objects:
public class MarketUpdatePool {
private final Queue<MarketUpdate> pool = new ConcurrentLinkedQueue<>();
public MarketUpdate acquire() {
MarketUpdate update = pool.poll();
return update != null ? update.reset() : new MarketUpdate();
}
public void release(MarketUpdate update) {
pool.offer(update);
}
}
Avoid boxing — use primitive collections:
// Bad — boxes every long to Long, creates pressure
Map<Long, Double> prices = new HashMap<>();
// Better — primitive long-to-double map (Eclipse Collections or Trove)
LongDoubleHashMap prices = new LongDoubleHashMap();
Reuse buffers in the streaming data path:
// Reuse ByteBuffer across JSON parses rather than allocating per message
private final byte[] parseBuffer = new byte[64 * 1024];
private final JsonParser parser = jsonFactory.createParser(parseBuffer);
The flags I use in production for a Betfair trading framework on a 16-core server with 32GB RAM:
# Java 21, ZGC, 8GB heap for the trading service
java \
-XX:+UseZGC \
-Xms4g -Xmx8g \
-XX:SoftMaxHeapSize=6g \
-XX:+ZGenerational \ # generational ZGC (Java 21+) — better throughput
-XX:+AlwaysPreTouch \ # pre-touch pages at startup — avoids OS page faults at runtime
-XX:+DisableExplicitGC \ # ignore System.gc() calls from libraries
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/var/dumps/ \
-Xlog:gc*:file=/var/log/gc.log:time,uptime:filecount=5,filesize=10m \
-jar trading-framework.jar
-XX:+ZGenerational is available in Java 21 and makes ZGC significantly more efficient by separating young and old generation collection — recommended for most applications.
jcmd <pid> GC.run + allocation profiling reveals the hottest allocation sites.-XX:+AlwaysPreTouch is worth it for long-running services. Without it, the OS lazily allocates physical pages, causing page fault latency spikes early in the application lifecycle. Pre-touch eliminates this at the cost of longer startup.If you’re looking for a Java contractor who knows this space inside out, get in touch.
]]>Smarkets provides two API surfaces:
api.smarkets.com/v3/) — market browsing, order management, account operationsAuthentication uses OAuth2 tokens. API access is available via the developer portal; you’ll need to apply for a trading account with API access.
Smarkets uses a session token obtained via login:
@Service
public class SmarketsAuthService {
private static final String AUTH_URL = "https://api.smarkets.com/v3/sessions/";
private final RestClient restClient;
public String login(String username, String password) {
LoginRequest loginRequest = new LoginRequest(username, password);
LoginResponse response = restClient.post()
.uri(AUTH_URL)
.contentType(MediaType.APPLICATION_JSON)
.body(loginRequest)
.retrieve()
.body(LoginResponse.class);
return response.token(); // bearer token for subsequent requests
}
}
Include the token in subsequent requests:
HttpHeaders headers = new HttpHeaders();
headers.setBearerAuth(token);
headers.setContentType(MediaType.APPLICATION_JSON);
Sessions have limited TTLs; refresh tokens before they expire, or handle 401 responses with automatic re-authentication.
public List<SmarketsEvent> getHorseRacingEvents() {
// List event types
String response = restClient.get()
.uri("/v3/events/?type=race&state=upcoming&sort=start_datetime")
.header("Authorization", "Bearer " + token)
.retrieve()
.body(String.class);
// Parse events from JSON response
return objectMapper.readValue(response, EventsResponse.class).getEvents();
}
public List<SmarketsMarket> getMarketsForEvent(String eventId) {
MarketsResponse response = restClient.get()
.uri("/v3/events/" + eventId + "/markets/")
.header("Authorization", "Bearer " + token)
.retrieve()
.body(MarketsResponse.class);
return response.getMarkets();
}
The Smarkets API uses string IDs throughout. A market has a contractId per runner — the equivalent of Betfair’s selectionId.
public List<Quote> getPrices(String marketId) {
QuotesResponse response = restClient.get()
.uri("/v3/markets/" + marketId + "/quotes/")
.header("Authorization", "Bearer " + token)
.retrieve()
.body(QuotesResponse.class);
return response.getQuotes();
}
Each Quote contains bid prices (equivalent to Betfair’s available-to-lay from the customer perspective) and offer prices (equivalent to available-to-back). The terminology maps to traditional financial markets — Smarkets was founded with a financial markets background and it shows.
public OrderResponse placeBackOrder(
String marketId, String contractId,
String price, String quantity) {
// Smarkets uses fractional odds represented as rationals (e.g. "4/1")
// or decimal expressed as numerator/denominator pairs
PlaceOrderRequest request = PlaceOrderRequest.builder()
.marketId(marketId)
.contractId(contractId)
.side("buy") // "buy" = back, "sell" = lay (financial convention)
.price(price) // e.g. "4.5" for decimal odds
.quantity(quantity) // stake in pence as a string, e.g. "1000" = £10
.type("limit")
.build();
return restClient.post()
.uri("/v3/orders/")
.header("Authorization", "Bearer " + token)
.contentType(MediaType.APPLICATION_JSON)
.body(request)
.retrieve()
.body(OrderResponse.class);
}
Note the buy/sell terminology — consistent with financial markets, opposite to Betfair’s BACK/LAY convention. buy means you want the selection to win; sell means you want it to lose.
Smarkets provides real-time price updates via WebSocket:
@Component
public class SmarketsStreamClient {
private static final String STREAM_URL = "wss://stream.smarkets.com/";
private WebSocketSession session;
public void connect(List<String> marketIds) throws Exception {
WebSocketClient client = new StandardWebSocketClient();
session = client.execute(new TextWebSocketHandler() {
@Override
public void handleTextMessage(WebSocketSession session, TextMessage message) {
processMessage(message.getPayload());
}
@Override
public void afterConnectionClosed(WebSocketSession session, CloseStatus status) {
log.warn("Smarkets stream closed: {}", status);
scheduleReconnect();
}
}, STREAM_URL).get();
// Subscribe to markets
SubscribeRequest sub = new SubscribeRequest(marketIds);
session.sendMessage(new TextMessage(objectMapper.writeValueAsString(sub)));
}
private void processMessage(String payload) {
// Parse market update and apply to local state cache
SmarketsMarketUpdate update = objectMapper.readValue(payload, SmarketsMarketUpdate.class);
stateCache.apply(update);
}
}
A common adapter interface normalises Smarkets and Betfair behind a shared model:
public interface ExchangeAdapter {
String exchangeId();
List<ExchangeMarket> findMarkets(ExchangeMarketFilter filter);
ExchangePriceView getPrices(String exchangeMarketId);
ExchangeOrderResult placeOrder(ExchangeOrder order);
void cancelOrder(String exchangeOrderId);
}
@Component
public class SmarketsAdapter implements ExchangeAdapter {
@Override
public String exchangeId() { return "SMARKETS"; }
@Override
public ExchangeOrderResult placeOrder(ExchangeOrder order) {
// Translate ExchangeOrder to Smarkets request
// Handle buy/sell convention mapping
String side = order.getSide() == Side.BACK ? "buy" : "sell";
OrderResponse response = smarketsClient.placeOrder(
order.getMarketId(), order.getSelectionId(),
String.valueOf(order.getPrice()), String.valueOf((int)(order.getStake() * 100))
);
return ExchangeOrderResult.fromSmarkets(response);
}
}
The strategy layer works only against ExchangeAdapter. Arbitrage opportunities between Betfair and Smarkets become detectable when both adapters report prices for correlated markets.
| Aspect | Smarkets | Betfair |
|---|---|---|
| Commission | 2% flat | 2–5% tiered |
| API style | REST + WebSocket | REST + TCP streaming |
| Streaming | WebSocket (standard) | Custom TCP/ESA protocol |
| Terminology | Financial (buy/sell) | Exchange (back/lay) |
| Liquidity | Lower overall, strong in politics/US sports | Highest in UK horse racing |
| Developer docs | Good | Excellent |
buy = back, sell = lay, bid = available-to-lay, offer = available-to-back — the financial-market naming is internally consistent but opposite to what Betfair developers expect.If you’re looking for a Java contractor who knows this space inside out, get in touch.
]]>Actuator exposes operational endpoints over HTTP. The most important for production is /actuator/health:
management:
endpoints:
web:
exposure:
include: health, info, metrics, prometheus, loggers
endpoint:
health:
show-details: when-authorized # don't expose internals publicly
show-components: when-authorized
health:
livenessState:
enabled: true
readinessState:
enabled: true
/actuator/health/liveness tells Kubernetes whether the process is alive (should it restart?). /actuator/health/readiness tells it whether the service is ready to receive traffic.
Custom health indicators let you express the health of dependencies:
@Component("betfairStream")
public class BetfairStreamHealthIndicator implements HealthIndicator {
private final BetfairStreamClient streamClient;
@Override
public Health health() {
if (!streamClient.isConnected()) {
return Health.down()
.withDetail("reason", "Streaming connection down")
.withDetail("lastConnected", streamClient.getLastConnectedAt())
.build();
}
Duration lastHeartbeat = streamClient.timeSinceLastHeartbeat();
if (lastHeartbeat.toSeconds() > 30) {
return Health.down()
.withDetail("reason", "No heartbeat for " + lastHeartbeat.toSeconds() + "s")
.build();
}
return Health.up()
.withDetail("subscriptions", streamClient.getSubscriptionCount())
.withDetail("lastHeartbeat", lastHeartbeat.toMillis() + "ms ago")
.build();
}
}
Micrometer is Spring Boot’s metrics abstraction. You write metrics once against the Micrometer API; the backend (Prometheus, Datadog, CloudWatch, etc.) is a dependency you swap.
The four core types:
@Service
public class MarketMetrics {
private final Counter betsPlaced;
private final Counter betsFailed;
private final Timer betPlacementLatency;
private final Gauge openPositions;
private final DistributionSummary stakeDistribution;
public MarketMetrics(MeterRegistry registry, PositionTracker tracker) {
this.betsPlaced = Counter.builder("trading.bets.placed")
.description("Total bets successfully placed")
.tag("exchange", "betfair")
.register(registry);
this.betsFailed = Counter.builder("trading.bets.failed")
.description("Total bet placement failures")
.tag("exchange", "betfair")
.register(registry);
this.betPlacementLatency = Timer.builder("trading.bet.placement.latency")
.description("Time from order decision to API response")
.publishPercentiles(0.5, 0.95, 0.99)
.register(registry);
this.openPositions = Gauge.builder("trading.positions.open", tracker,
PositionTracker::getOpenPositionCount)
.description("Number of currently open positions")
.register(registry);
this.stakeDistribution = DistributionSummary.builder("trading.stake.distribution")
.description("Distribution of bet stakes in pence")
.baseUnit("pence")
.publishPercentiles(0.5, 0.95, 0.99)
.register(registry);
}
}
Use tags to slice metrics by meaningful dimensions:
// Record bet placement with outcome tags
Timer.Sample sample = Timer.start();
try {
PlaceExecutionReport report = betfairClient.placeOrders(ssoid, request);
betsPlaced.increment();
stakeDistribution.record(request.getTotalStakePence());
sample.stop(betPlacementLatency.tag("result", "success").register(registry));
} catch (Exception e) {
betsFailed.increment();
sample.stop(betPlacementLatency.tag("result", "failure").register(registry));
}
Add the Prometheus dependency:
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
Expose the scrape endpoint:
management:
endpoints:
web:
exposure:
include: prometheus
metrics:
export:
prometheus:
enabled: true
Configure Prometheus to scrape:
# prometheus.yml
scrape_configs:
- job_name: 'trading-service'
metrics_path: '/actuator/prometheus'
scrape_interval: 15s
static_configs:
- targets: ['trading-service:8080']
Technical metrics (JVM heap, HTTP request rate, database pool size) are valuable but not sufficient. The dashboards I find most useful in production combine technical metrics with business metrics:
// Kafka consumer lag — technical + business signal
Gauge.builder("kafka.consumer.lag", consumerLag, ConsumerLagProvider::getTotalLag)
.tags("topic", "claim-events", "consumer-group", "claims-processor")
.register(registry);
// Daily P&L — business metric
Gauge.builder("trading.daily.pnl.pence", pnlTracker, PnlTracker::getDailyPnlPence)
.register(registry);
// Kill switch state — operational metric
Gauge.builder("trading.kill.switch.active",
riskController, rc -> rc.isKillSwitchActive() ? 1.0 : 0.0)
.register(registry);
In Grafana, set alerts on:
trading.kill.switch.active == 1 → page immediatelykafka.consumer.lag > 1000 for a sustained 5 minutes → investigatetrading.bets.failed rate > 5% → investigatejvm.memory.used / jvm.memory.max > 0.85 → memory pressureFor request flows that span multiple services, add distributed tracing:
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-reporter-brave</artifactId>
</dependency>
management:
tracing:
sampling:
probability: 0.1 # sample 10% of requests
Traces propagate automatically across HTTP calls and Kafka messages in Spring Boot 3.x. The traceId appears in structured log output, linking logs to the trace in Zipkin or Grafana Tempo.
@Timed for quick wins. Spring’s @Timed annotation on @RequestMapping methods and @KafkaListener handlers instruments latency automatically without boilerplate.management.server.port=8090 to expose Actuator on a separate port, accessible only within your VPC. Don’t expose internal health details to the internet.If you’re looking for a Java contractor who knows this space inside out, get in touch.
]]>A Betfair market moves through several states, visible in the Streaming API’s MarketDefinition:
public enum MarketStatus {
INACTIVE, // market exists but not yet accepting bets
OPEN, // pre-race, accepting bets, match-time clock running
SUSPENDED, // temporarily suspended (VAR check, injury, etc.)
CLOSED // market settled
}
The transition that matters most for trading architecture is OPEN → IN_PLAY. This happens at the event start and is signalled in the streaming API via MarketDefinition.inPlay = true.
public void onMarketDefinitionChange(String marketId, MarketDefinition definition) {
MarketState current = markets.get(marketId);
if (current == null) return;
boolean wasInPlay = current.isInPlay();
boolean isNowInPlay = Boolean.TRUE.equals(definition.isInPlay());
if (!wasInPlay && isNowInPlay) {
onMarketGoesInPlay(marketId);
}
if (definition.getStatus() == MarketStatus.SUSPENDED) {
onMarketSuspended(marketId, wasInPlay);
}
current.update(definition);
}
The onMarketGoesInPlay event is your trigger to switch from pre-race mode (signal-based, deliberate entry) to in-play mode (position management, hedging, emergency exits).
Pre-race trading has time on its side. Your analytical loop runs every 500ms–1s, evaluating WoM, LTP, and OFI signals. The latency requirements are lenient — a 50ms delay in signal processing rarely changes the outcome.
@Component
public class PreRaceStrategyEngine {
private final Map<String, MarketSignalBuffer> signalBuffers = new ConcurrentHashMap<>();
// Called on each Streaming API delta
public void onMarketUpdate(String marketId, MarketChangeMessage mcm) {
MarketSignalBuffer buffer = signalBuffers.get(marketId);
if (buffer == null) return;
buffer.addDelta(mcm);
// Evaluate at most once per 500ms
if (buffer.shouldEvaluate()) {
MarketSignals signals = signalCalculator.calculate(buffer.getState());
strategies.forEach(s -> s.onSignalUpdate(context(marketId), signals));
}
}
public void onMarketGoesInPlay(String marketId) {
// Stop pre-race evaluation
signalBuffers.remove(marketId);
// Transfer to in-play engine
inPlayEngine.trackMarket(marketId);
}
}
The MarketSignalBuffer rate-limits evaluation — no point running signal calculations on every Streaming API tick when the signals require 20-observation windows to be meaningful.
In-play is a different beast. Prices move at sports event pace — a goal in football can move a selection from 2.5 to 1.2 in under a second. The in-play engine must respond in milliseconds, not seconds. In-play strategy is primarily about position management:
@Component
public class InPlayPositionManager {
private final Map<String, InPlayPosition> openPositions = new ConcurrentHashMap<>();
private final OrderManager orderManager;
private final RiskController riskController;
// Called on every Streaming API delta in-play
public void onInPlayUpdate(String marketId, MarketChangeMessage mcm) {
InPlayPosition position = openPositions.get(marketId);
if (position == null) return;
for (RunnerChange rc : mcm.getMarketChanges().get(0).getRunnerChanges()) {
double currentLtp = extractLtp(rc);
position.updateLtp(rc.getId(), currentLtp);
// Check hedge conditions
if (shouldHedge(position, rc.getId(), currentLtp)) {
hedgePosition(marketId, position, rc.getId(), currentLtp);
}
}
}
private boolean shouldHedge(InPlayPosition position, long selectionId, double currentLtp) {
double entryPrice = position.getEntryPrice(selectionId);
double currentPnl = position.calculatePnl(selectionId, currentLtp);
// Hedge if profit target hit
if (currentPnl >= position.getProfitTarget()) return true;
// Hedge if stop loss hit
if (currentPnl <= position.getStopLoss()) return true;
return false;
}
}
The critical difference: no signal buffering in-play. Every delta is evaluated immediately.
Betfair applies tighter API rate limits in-play. The REST API allows around 5 calls per second per application key for order operations in-play — roughly half the pre-race limit in some markets. This is why in-play order management must be batched: multiple position hedges in the same API call wherever possible:
public void hedgeMultiplePositions(String marketId, List<HedgeInstruction> hedges) {
List<PlaceInstruction> instructions = hedges.stream()
.map(h -> buildLayInstruction(h.selectionId(), h.price(), h.size()))
.toList();
// One API call for all hedges
PlaceExecutionReport report = orderManager.placeOrders(marketId, instructions);
processReport(report, hedges);
}
Suspension is particularly dangerous in-play. When a market suspends, open positions can’t be hedged, but liability from unmatched lays remains. Your architecture must:
MarketDefinition.status = SUSPENDED)public void onMarketSuspended(String marketId, boolean wasInPlay) {
log.warn("Market {} suspended (in-play: {})", marketId, wasInPlay);
if (wasInPlay) {
// In-play suspension — prices will change on resume
InPlayPosition position = openPositions.get(marketId);
if (position != null) {
position.flagSuspended();
// Attempt cancel but expect potential failure
try {
orderManager.cancelAllOpenOrders(marketId);
} catch (Exception e) {
log.error("Cancel on suspension failed — will retry on resume", e);
position.setPendingCancel(true);
}
}
}
}
public void onMarketResumed(String marketId) {
InPlayPosition position = openPositions.get(marketId);
if (position == null) return;
position.clearSuspended();
if (position.isPendingCancel()) {
// Retry cancel now market is live again
orderManager.cancelAllOpenOrders(marketId);
}
// Reconcile — prices may have moved significantly during suspension
orderManager.reconcileOpenOrders(marketId);
}
PersistenceType.LAPSE on pre-race orders — they’ll cancel at the off unless you explicitly convert them. An accidental in-play position at bad odds can ruin a profitable session.If you’re looking for a Java contractor who knows this space inside out, get in touch.
]]>Lambda cold start time breaks down into:
Step 3 is the main culprit. Spring Boot on the JVM needs to load and initialise hundreds of beans, run auto-configuration, and connect to external dependencies. On a warmed Lambda with an already-running JVM, this cost is zero. On a cold start, you pay it in full.
Before optimising, measure. Add the AWS Lambda Powertools Java dependency for structured metrics:
public class ClaimHandler implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {
private final ClaimService claimService;
public ClaimHandler() {
// This constructor runs at cold start
long start = System.currentTimeMillis();
this.claimService = SpringLambda.getBean(ClaimService.class);
long initTime = System.currentTimeMillis() - start;
log.info("Cold start init time: {}ms", initTime);
}
@Override
public APIGatewayProxyResponseEvent handleRequest(
APIGatewayProxyRequestEvent event, Context context) {
// Warm invocations arrive here directly
return claimService.handle(event);
}
}
Log and track initTime. Track in CloudWatch and alert when p99 cold start time exceeds your threshold.
AWS Lambda SnapStart (available for Java 11+) snapshots the Lambda execution environment after initialisation and restores from the snapshot on subsequent cold starts. For Spring Boot applications, this means you pay the initialisation cost once — future cold starts restore from the snapshot in ~200ms.
Enable it in your template.yml:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
ClaimFunction:
Type: AWS::Serverless::Function
Properties:
Runtime: java21
SnapStart:
ApplyOn: PublishedVersions
Handler: com.example.ClaimHandler
CodeUri: target/claim-service.jar
MemorySize: 512
Timeout: 30
SnapStart requires that your initialisation code is idempotent — the snapshot is taken once and restored many times. Watch for:
Implement CRaC hooks to handle pre-checkpoint and post-restore lifecycle:
import org.crac.Context;
import org.crac.Core;
import org.crac.Resource;
@Component
public class SnapStartLifecycle implements Resource {
private final DataSource dataSource;
@PostConstruct
public void register() {
Core.getGlobalContext().register(this);
}
@Override
public void beforeCheckpoint(Context<? extends Resource> ctx) throws Exception {
// Close connections before snapshot is taken
dataSource.getConnection().close();
}
@Override
public void afterRestore(Context<? extends Resource> ctx) throws Exception {
// Re-establish connections after restore
dataSource.getConnection(); // warm up connection pool
}
}
GraalVM compiles your Spring Boot application to a native executable — no JVM startup, AOT-compiled code. Cold starts of 50–200ms are achievable:
<plugin>
<groupId>org.graalvm.buildtools</groupId>
<artifactId>native-maven-plugin</artifactId>
<configuration>
<imageName>claim-function</imageName>
<buildArgs>
<buildArg>--no-fallback</buildArg>
<buildArg>--enable-url-protocols=https</buildArg>
</buildArgs>
</configuration>
</plugin>
Spring Boot 3.x has first-class GraalVM support via spring-boot-starter + AOT processing. Most Spring features work — but dynamic features (reflection, proxies, runtime classpath scanning) require explicit hint configuration:
@RegisterReflectionForBinding({ClaimEvent.class, ClaimResponse.class})
@Configuration
public class LambdaConfig {
// beans for Lambda handler
}
Native image compilation is slow (5–10 minutes on a typical build machine) and imposes constraints on dynamic features. For Lambda functions with stable, well-defined inputs and outputs, it’s the best option for cold start performance.
Without native image, JVM flags reduce cold start time:
# In Lambda function environment variables
JAVA_TOOL_OPTIONS="-XX:+TieredCompilation -XX:TieredStopAtLevel=1"
TieredStopAtLevel=1 disables C2 (optimising) compilation and uses only the fast interpreter + C1 (baseline) compiler. Startup is faster; peak throughput is lower. For Lambda where invocations are short-lived, this is generally the right trade-off.
Lambda CPU allocation scales with memory. A 512MB Lambda gets half a vCPU; 1024MB gets a full vCPU; 1769MB gets two vCPUs. For a Spring Boot application:
Extra memory reduces cold start time faster than the marginal cost increase. For a Lambda behind an API Gateway, 1024–2048MB is often optimal.
Lambda is not appropriate for:
Lambda is ideal for: event-driven processing triggered by S3, SQS, SNS, EventBridge; infrequent API endpoints; scheduled batch jobs; and data transformation functions.
BeanDefinitionCustomizerParameterizedCondition and @Bean-based configuration start faster than classpath component scanning.If you’re looking for a Java contractor who knows this space inside out, get in touch.
]]>CompletableFuture and one failed, the other two kept running — consuming resources, potentially making external calls, and potentially producing results that would never be used. Structured concurrency, finalised in Java 21, gives you a disciplined model: tasks are scoped, they have clear lifetimes, and the scope doesn’t exit until all tasks have finished or been cancelled.
Structured concurrency enforces a simple invariant: a task’s lifetime is bounded by the scope that created it. The scope doesn’t complete until all forked tasks have completed. If you exit the scope with tasks still running, they’re cancelled. This eliminates the most common class of thread leak in concurrent Java.
StructuredTaskScope is the central API:
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
var runnerTask = scope.fork(() -> fetchRunners(marketId));
var priceTask = scope.fork(() -> fetchPrices(marketId));
var formTask = scope.fork(() -> fetchForm(marketId));
scope.join(); // wait for all tasks to complete
scope.throwIfFailed(); // propagate any exceptions
return new MarketView(
runnerTask.get(),
priceTask.get(),
formTask.get()
);
} // scope closes here — all tasks guaranteed done or cancelled
Three fetches run concurrently. If any one throws, scope.join() returns, throwIfFailed() throws the exception, and the try-with-resources close cancels the remaining tasks. No dangling threads.
ShutdownOnFailure shuts down the scope as soon as any task fails, then cancels remaining tasks:
public EnrichedMarket enrichMarket(String marketId) throws Exception {
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
Subtask<RunnerData> runners = scope.fork(() -> runnerService.fetch(marketId));
Subtask<FormData> form = scope.fork(() -> formService.fetch(marketId));
Subtask<WeatherData> weather = scope.fork(() -> weatherService.fetch(marketId));
scope.join().throwIfFailed(); // join and propagate first failure
return EnrichedMarket.of(runners.get(), form.get(), weather.get());
}
}
Subtask.get() is safe to call after throwIfFailed() — all tasks either succeeded (state = SUCCESS) or were cancelled. Calling get() on a cancelled subtask throws IllegalStateException — which is why you check the scope result before calling get().
If you need a timeout:
scope.joinUntil(Instant.now().plusSeconds(5)); // deadline, not duration
scope.throwIfFailed();
If the deadline passes before all tasks finish, the scope cancels remaining tasks and throwIfFailed() throws a TimeoutException.
ShutdownOnSuccess completes as soon as the first task succeeds, cancels the rest:
public MarketData fetchFromFastestSource(String marketId) throws Exception {
try (var scope = new StructuredTaskScope.ShutdownOnSuccess<MarketData>()) {
scope.fork(() -> primarySource.fetch(marketId));
scope.fork(() -> secondarySource.fetch(marketId));
scope.fork(() -> tertiarySource.fetch(marketId));
scope.join();
return scope.result(); // returns the first successful result
}
}
This is a redundant-fetch pattern: three providers are queried simultaneously, and the fastest successful response wins. The others are cancelled. It’s useful when you need low latency and have multiple data sources of varying reliability.
The classic CompletableFuture leak pattern:
// Old — if processA fails, processB keeps running and nobody notices
CompletableFuture<ResultA> a = CompletableFuture.supplyAsync(() -> processA());
CompletableFuture<ResultB> b = CompletableFuture.supplyAsync(() -> processB());
ResultA ra = a.join(); // throws if a failed
ResultB rb = b.join(); // b is still running, but we've already thrown
With structured concurrency:
// New — if processA fails, processB is cancelled before the scope exits
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
var a = scope.fork(() -> processA());
var b = scope.fork(() -> processB());
scope.join().throwIfFailed();
return process(a.get(), b.get());
}
// b is guaranteed done or cancelled here
The scope is the mechanism that provides the guarantee. As a AutoCloseable, it’s compatible with try-with-resources, and the close operation waits for tasks to terminate.
For more complex fan-out/fan-in scenarios, extend StructuredTaskScope with a custom policy:
public class MajorityResultScope<T> extends StructuredTaskScope<T> {
private final List<T> results = Collections.synchronizedList(new ArrayList<>());
private final int required;
public MajorityResultScope(int required) {
this.required = required;
}
@Override
protected void handleComplete(Subtask<? extends T> subtask) {
if (subtask.state() == Subtask.State.SUCCESS) {
results.add(subtask.get());
if (results.size() >= required) {
shutdown(); // enough results — cancel remaining
}
}
}
public List<T> results() { return List.copyOf(results); }
}
Both have their place in Java 21:
| Concern | Structured Concurrency | CompletableFuture |
|---|---|---|
| Code readability | Sequential, clear | Callback chains, dense |
| Thread leak risk | Impossible by design | Requires careful handling |
| Exception propagation | Clean, first failure wins | Wrapped in CompletionException |
| Complex async pipelines | Cumbersome | Natural (thenCompose, etc.) |
| Debugging | Standard thread dumps | Async stack traces are poor |
| Java version | Java 21+ (stable) | Java 8+ |
I use structured concurrency for fan-out patterns where I need all results (or the first result) and clean cancellation behaviour. I still use CompletableFuture for complex reactive pipelines where thenCompose chains read more naturally than nested scopes.
StructuredTaskScope.fork() creates virtual threads by default in Java 21. The combination gives you millions of lightweight concurrent tasks with disciplined scoping.CompletableFuture.allOf. Wrapping CompletableFuture inside a structured scope defeats the purpose — the futures run independently of the scope’s lifecycle.Thread.currentThread().isInterrupted() or use interruptible IO — otherwise they won’t cancel promptly.If you’re looking for a Java contractor who knows this space inside out, get in touch.
]]>