Running a trading strategy live without backtesting it first is gambling in both senses. Betfair provides historical market data files that contain every price movement and traded volume for every market ever run on the exchange — a complete record you can replay to simulate how a strategy would have performed.
This post covers the mechanics of building a backtesting engine in Java: parsing Betfair’s historical stream format, constructing a simulated order book, executing hypothetical orders, and computing meaningful performance metrics.
The Betfair historical data format
Betfair’s historical data is distributed as bzip2-compressed files, one file per market. Each file is a sequence of newline-delimited JSON objects — the same format emitted by the Betfair Streaming API — so a replay engine and a live streaming client share most of their parsing code.
Each line is a market change message:
{
"op": "mcm",
"clk": "AAAAAAAA",
"pt": 1716800000000,
"mc": [
{
"id": "1.234567890",
"rc": [
{
"id": 12345678,
"batb": [[0, 1.98, 250.0]],
"batl": [[0, 2.02, 180.0]],
"trd": [[1.98, 1200.0]],
"ltp": 2.0,
"tv": 8500.0
}
]
}
]
}
batb — best available to back (sorted by price descending); batl — best available to lay; trd — traded volume at each price; ltp — last traded price; tv — total volume matched.
The array [0, 1.98, 250.0] means: at ladder level 0, price 1.98, £250 available.
Data model
public record MarketSnapshot(
String marketId,
long publishTime,
Map<Long, RunnerSnapshot> runners
) {}
public record RunnerSnapshot(
long selectionId,
List<PriceSize> bestBack, // sorted highest first
List<PriceSize> bestLay, // sorted lowest first
List<PriceSize> traded,
double lastTradedPrice,
double totalMatched
) {}
public record PriceSize(double price, double size) {}
Stream parser
The parser reads the compressed file line by line, applies incremental updates to a mutable market state, and emits immutable snapshots:
public class BetfairStreamParser {
private final ObjectMapper mapper = new ObjectMapper();
private final Map<String, MutableMarketState> markets = new HashMap<>();
public Stream<MarketSnapshot> parse(Path dataFile) throws IOException {
var input = new BZip2CompressorInputStream(
new BufferedInputStream(Files.newInputStream(dataFile)));
return new BufferedReader(new InputStreamReader(input))
.lines()
.filter(line -> !line.isBlank())
.flatMap(line -> applyAndEmit(line).stream());
}
private List<MarketSnapshot> applyAndEmit(String line) {
try {
var node = (ObjectNode) mapper.readTree(line);
if (!"mcm".equals(node.path("op").asText())) return List.of();
long pt = node.path("pt").asLong();
var snapshots = new ArrayList<MarketSnapshot>();
for (var mc : node.path("mc")) {
var marketId = mc.path("id").asText();
var state = markets.computeIfAbsent(marketId, MutableMarketState::new);
state.apply(mc, pt);
snapshots.add(state.snapshot());
}
return snapshots;
} catch (Exception e) {
return List.of();
}
}
}
MutableMarketState holds the current ladder for each runner and applies delta updates. Betfair sends full ladder state on the first message, then incremental changes thereafter.
class MutableMarketState {
private final String marketId;
private long publishTime;
private final Map<Long, MutableRunnerState> runners = new HashMap<>();
void apply(JsonNode mc, long pt) {
this.publishTime = pt;
for (var rc : mc.path("rc")) {
long selId = rc.path("id").asLong();
var runner = runners.computeIfAbsent(selId, MutableRunnerState::new);
runner.apply(rc);
}
}
MarketSnapshot snapshot() {
var runnerSnapshots = runners.entrySet().stream()
.collect(Collectors.toMap(Map.Entry::getKey,
e -> e.getValue().snapshot()));
return new MarketSnapshot(marketId, publishTime, runnerSnapshots);
}
}
Simulated order execution
In a backtest you can’t send orders to the exchange. Instead you simulate order matching: a back bet matches against available lay liquidity, and vice versa.
public class SimulatedOrderBook {
private final List<SimulatedOrder> pending = new ArrayList<>();
private final List<ExecutedBet> executed = new ArrayList<>();
public void submitBack(long selectionId, double price, double stake, long time) {
pending.add(new SimulatedOrder(selectionId, price, stake, Side.BACK, time));
}
public void processSnapshot(MarketSnapshot snapshot) {
var iterator = pending.iterator();
while (iterator.hasNext()) {
var order = iterator.next();
var runner = snapshot.runners().get(order.selectionId());
if (runner == null) continue;
// A back bet at price P executes if lay liquidity exists at <= P
if (order.side() == Side.BACK) {
double availableLay = runner.bestLay().stream()
.filter(ps -> ps.price() <= order.price())
.mapToDouble(PriceSize::size)
.sum();
if (availableLay >= order.stake()) {
executed.add(new ExecutedBet(order, snapshot.publishTime()));
iterator.remove();
}
}
}
}
public List<ExecutedBet> executed() { return List.copyOf(executed); }
}
This is a first-price, best-effort model. Real Betfair matching follows a strict price-time priority queue — for a more accurate simulation you’d track queue position and account for partial fills.
Strategy interface
Strategies receive each market snapshot and emit orders:
@FunctionalInterface
public interface TradingStrategy {
List<OrderInstruction> onSnapshot(MarketSnapshot snapshot, PortfolioState portfolio);
}
public record OrderInstruction(
long selectionId,
Side side,
double price,
double stake
) {}
A simple lay-the-favourite strategy:
public class LayTheFavouriteStrategy implements TradingStrategy {
private static final double MAX_PRICE = 3.5;
private static final double STAKE = 10.0;
private final Set<String> positioned = new HashSet<>();
@Override
public List<OrderInstruction> onSnapshot(MarketSnapshot snapshot, PortfolioState portfolio) {
if (positioned.contains(snapshot.marketId())) return List.of();
return snapshot.runners().values().stream()
.filter(r -> !r.bestBack().isEmpty())
.min(Comparator.comparingDouble(RunnerSnapshot::lastTradedPrice))
.filter(fav -> fav.lastTradedPrice() <= MAX_PRICE
&& fav.lastTradedPrice() > 1.01)
.map(fav -> {
positioned.add(snapshot.marketId());
double layPrice = fav.bestLay().isEmpty()
? fav.lastTradedPrice()
: fav.bestLay().get(0).price();
return List.of(new OrderInstruction(
fav.selectionId(), Side.LAY, layPrice, STAKE));
})
.orElse(List.of());
}
}
The backtest runner
public class BacktestRunner {
private final BetfairStreamParser parser;
private final TradingStrategy strategy;
private final SimulatedOrderBook orderBook;
public BacktestResult run(List<Path> dataFiles) throws IOException {
var portfolio = new PortfolioState();
for (Path file : dataFiles) {
orderBook.clear();
try (var snapshots = parser.parse(file)) {
snapshots.forEach(snapshot -> {
orderBook.processSnapshot(snapshot);
var instructions = strategy.onSnapshot(snapshot, portfolio);
instructions.forEach(inst ->
orderBook.submitBack(inst.selectionId(),
inst.price(), inst.stake(), snapshot.publishTime()));
});
}
orderBook.executed().forEach(portfolio::record);
}
return BacktestResult.from(portfolio);
}
}
Performance metrics
public class BacktestResult {
private final List<ExecutedBet> bets;
public double totalPnl() {
return bets.stream().mapToDouble(ExecutedBet::pnl).sum();
}
public double roi() {
double totalStaked = bets.stream().mapToDouble(b -> b.order().stake()).sum();
return totalStaked == 0 ? 0 : totalPnl() / totalStaked;
}
public double strikeRate() {
long winners = bets.stream().filter(b -> b.pnl() > 0).count();
return bets.isEmpty() ? 0 : (double) winners / bets.size();
}
public double sharpeRatio() {
var pnls = bets.stream().mapToDouble(ExecutedBet::pnl).toArray();
double mean = Arrays.stream(pnls).average().orElse(0);
double variance = Arrays.stream(pnls)
.map(p -> Math.pow(p - mean, 2))
.average().orElse(0);
double stdDev = Math.sqrt(variance);
return stdDev == 0 ? 0 : mean / stdDev;
}
public double maxDrawdown() {
double peak = 0, drawdown = 0, running = 0;
for (ExecutedBet bet : bets) {
running += bet.pnl();
if (running > peak) peak = running;
drawdown = Math.min(drawdown, running - peak);
}
return drawdown;
}
}
A positive Sharpe ratio indicates the strategy earns more per unit of volatility than it loses. Max drawdown tells you the worst peak-to-trough loss — the number that tests your nerve in live trading.
Walk-forward testing
A backtest that optimises parameters on the full dataset is overfitted. Walk-forward testing trains on a window of historical data and tests on the next unseen period:
public class WalkForwardTest {
public List<PeriodResult> run(List<Path> allFiles,
int trainMonths, int testMonths,
StrategyFactory factory) throws IOException {
var results = new ArrayList<PeriodResult>();
int i = 0;
while (i + trainMonths + testMonths <= allFiles.size()) {
var trainFiles = allFiles.subList(i, i + trainMonths);
var testFiles = allFiles.subList(i + trainMonths, i + trainMonths + testMonths);
var params = optimise(trainFiles, factory);
var strategy = factory.create(params);
var result = new BacktestRunner(parser, strategy, new SimulatedOrderBook())
.run(testFiles);
results.add(new PeriodResult(i, result));
i += testMonths;
}
return results;
}
}
If the strategy performs consistently across multiple out-of-sample periods, it has some evidence of robustness. If it only works on the training window, it’s curve-fitted to historical noise.
Practical limitations to keep in mind
Liquidity assumption. The simulator assumes your orders fill if sufficient opposing liquidity exists. In reality, large orders move the market — especially pre-race when liquidity thins. Reduce simulated stakes to account for market impact.
Timing. Historical data timestamps have millisecond resolution but replay is bounded by parsing speed. This matters for strategies that trade in the final seconds before off.
Commission. Betfair charges 5% on net winnings per market (lower for Premium Charge payers). Always deduct commission from P&L before drawing conclusions — a 3% ROI gross can become negative net.
Selection bias. Betfair’s historical archive goes back years and covers cancelled markets, abandoned races, and walkover results. Filter these or your win/loss ratio will be distorted.
A backtesting engine is only as useful as the data it runs on and the scepticism you apply to its results — treat it as evidence, not proof. If you’re building Betfair trading infrastructure and need a second opinion, hire me.
