The Betfair Historical Data Portal hands you the raw streaming feed back: every change message the API published for a market, one JSON object per line, compressed. It is the closest thing to replaying the market — which makes it the best raw material for backtesting. But it is not a spreadsheet. It is a sequence of deltas you have to reconstruct into state, and that reconstruction is where parsing goes wrong. This post is about the file format and the parsing discipline around it. You can see the whole thing working, 100% in the browser, in the /betfair-history-parser/ demo.
The file format
A history file is NDJSON — newline-delimited JSON — compressed with .bz2 on download. Decompress it first (bunzip2 on the command line, 7-Zip or WinRAR on Windows); the parser then reads plain text, one message per line:
{"op":"mcm","id":1,"clk":"SA1","pt":1747310400000,"ct":"SUB_IMAGE","mc":[{"id":"1.999001","img":true,"tv":22000,"marketDefinition":{"status":"OPEN","inPlay":false,"runners":[{"id":10001,"name":"Quantum Leap","sortPriority":1,"status":"Active"}]},"rc":[{"id":10001,"ltp":4.5,"tv":8800,"trd":[[4.5,4000],[4.6,2500]]}]}]}
The fields that matter:
- op — the operation;
mcm(market change message) is the one you will see in these files. - pt — publish time, epoch milliseconds UTC.
- clk — change key, a sequence marker used for reconciliation on live streams; in a file it lets you detect gaps.
- mc — the market changes. Each entry has a market id and carries whichever of these changed:
tv(total matched),marketDefinition(status, in-play flag, runner roster) andrc(runner changes). - img: true — this message is a full market image, not a delta. The state of the market resets to exactly what this message contains. The first message of a file is almost always a
SUB_IMAGE; from that point on you are tracking changes.
Deltas, not snapshots
The feed is delta-based by design. A runner change only carries what moved — ltp (last traded price), tv (total matched) and trd (traded-volume ticks as price/size pairs). To know what a runner was doing at time T you have to have applied every message before T. Two rules fall out:
- Never skip messages. The file is self-contained — the image arrives first — but a parser that drops messages rebuilds a wrong market from the first dropped line onwards.
- No
rcfor a runner means no change. The old state stands. Treating an absent field as a reset is the classic bug.
What you can extract
From one file you can rebuild everything the strategy saw: per-runner price history (LTP over time), every trade tick, total matched volume, the market’s timeline (OPEN → SUSPENDED → OPEN in-play → CLOSED), runner statuses (Active, Winner, Loser, Removed), and at the close the BSP and any reduction factors. The demo exports exactly this as three CSVs — a price-history file, a trade-ticks file and a runner summary — which is the raw material the backtesting and analysis posts build on.
The traps
- Compression — the portal serves
.bz2; a parser pointed at the compressed bytes fails confusingly. Decompress first, or stream through a decompressor. - Timestamps — epoch milliseconds UTC. Convert explicitly; treating them as local time or seconds shifts every chart.
- Price and size semantics — prices are decimal odds, sizes are in pounds. A
trdpair of[4.5, 4000]is £4,000 matched at 4.5, not 4,000 shares. - File size — these files are large. Read line by line and update state incrementally; slurping the whole file into memory is the most common way to make a “quick” parser unusable.
- The in-play transition —
marketDefinitionchanges mid-file, so “is this market in-play at time T” has to be tracked, not assumed. - Absent BSP — BSP appears only at the close and not on every market. Handle its absence rather than failing.
- Suspensions — a suspension message often carries no
rcat all. It is a state change, not a price change; the timeline has to record it even though no runner moved.
Why the discipline matters
Historical data is how you answer “would this strategy actually have worked” — the replay and backtest posts (market data replay, backtesting framework) depend on the feed being reconstructed faithfully. The parser is the front door: garbage in the state reconstruction, garbage in every backtest that follows. The discipline — apply images first, track state through deltas, handle transitions and absent fields — is exactly the same one a live streaming client needs (getting started, subscriptions), minus the network. If you can parse a history file correctly, the live feed is a solved problem in disguise.
If you’re building on Betfair historical data and want the parsing and analysis right, discuss your project.
