Case study · Distributed systems

Cutting a trading platform's recovery from 90+ minutes to 7

A Tier-1, multi-region equities platform took an hour and a half to come back from a restart. The fix wasn't a bigger box — it was reading Cassandra's internals and letting the data load the way it was already laid out.

Ritesh Sharma · · 4 min read
distributed-systemscassandraapache-igniteperformancefintechsre

Trading platforms have an unforgiving property: the cost of being down is a function of when you're down. An intraday restart during market hours isn't an inconvenience — it's exposure, with operational and regulatory weight behind every minute. The platform I work on is a multi-region system backing institutional equities flow, and when it restarted, one number dominated the war-room conversation: over 90 minutes before caches were warm enough to serve.

This is the story of taking that number to about seven minutes — and why most of the win came not from new infrastructure, but from noticing that the data's own layout had been offering us parallelism the whole time.

Where the time went

The platform keeps its working set in Apache Ignite caches, populated at startup from Cassandra. Profiling the restart made the bottleneck embarrassingly clear: cache population was, in effect, a single-threaded full-table scan — one loader walking the entire keyspace through one connection, row after row, while every component upstream waited.

−2⁶³ / 2⁶³−1 −2⁶² 0 2⁶² CASSANDRA token ring IGNITE CACHES warming · one stream warm after 90+ min LOADER startup 1 thread · 1 connection SELECT * FROM trades; // no WHERE clause −2⁶³ 2⁶³−1 dependent components COMPONENT Awaitingidle 90+ min COMPONENT Bwaitingidle 90+ min COMPONENT Cwaitingidle 90+ min WARM-UP 0 15 30 45 60 75 90 min 90+ MIN

Ninety minutes of serial I/O, in a system whose storage layer is built for concurrency. The question wasn't "how do we make the scan faster" — it was "why is there only one scan?"

The parallelism the data already wanted

Cassandra doesn't store a table as one long file. It hashes every partition key — Murmur3, by default — onto a ring of token values, and spreads ownership of that ring across the cluster. Which means the keyspace comes pre-divided: pick any set of token boundaries, and the ranges between them are contiguous, disjoint, and together cover everything. The database's own distribution mechanism is a free work-partitioning scheme.

TOKEN RING QUERY 6 ranges QUEUES default limit limit raised READERS 1 per range IGNITE partitions partition key token · 64-bit 'T-1041' MURMUR3 −3 968 589 391 818 315 427 SELECT * FROM trades WHERE token(trade_id) > ? AND token(trade_id) <= ? −2⁶³ 2⁶³−1 −6.1·10¹⁸ −3.1·10¹⁸ 0 3.1·10¹⁸ 6.1·10¹⁸ READER 0 READER 1 READER 2 READER 3 READER 4 READER 5 SERIAL PARALLEL 90+ MIN ≈ 7 MIN · 13× FASTER 0 15 30 45 60 75 90 min

So the load became that: the token space divided into mutually exclusive, collectively exhaustive ranges, one reader per range, all running concurrently. No coordination needed between readers — disjointness is guaranteed by construction, coverage by arithmetic. The remaining work was making Cassandra comfortable receiving that burst: raising queue capacities and tuning thresholds so a cluster sized for steady-state traffic would absorb a deliberate thundering herd at boot and shed none of it.

The lesson that generalizes: before adding infrastructure to go faster, check whether your storage layer's own partitioning scheme is already a parallel work queue. It usually is.

The other half: topology

Parallel loading attacked the biggest constant, but restarts were also unstable. The culprit was cache placement. Several caches were logically owned by specific components, yet Ignite distributes cache partitions dynamically across whatever nodes are in the ring — so as nodes trickled in during a restart, partitions kept migrating to each new arrival. The cluster spent its most fragile minutes reshuffling data it had only just loaded.

RING 3 NODES 4 NODES 5 NODES6 NODES 4 NODES5 NODES6 NODES PARTITIONS1024 REBALANCES 0 1 23 0 1 cell ≈ 8 partitions AFFINITYowner(p) = rendezvous(p,ring){A, B, C}) NODE FILTER starting deferred starting deferred starting deferred NODE A 342 parts 342 parts NODE B 341 parts 341 parts NODE C 341 parts 341 parts p417 p417 NODE D 0 parts NODE E 0 parts NODE F 0 parts receiving 256 parts 256 parts 256 parts 256 parts p417 receiving 205 parts 205 parts 205 parts 205 parts 204 parts p417 receiving 171 parts 171 parts 171 parts 171 parts 170 parts 170 parts p417 JOINSPOPULATE A B C D E F critical window · ring holds at 3 nodes

Two changes stopped the churn. First, node filters: component-owned caches were pinned to a designated core set of nodes, so partition placement stopped depending on who happened to be in the ring. Second, startup sequencing: components that host no cache data now join only after the core is up and populated. During the critical window, the ring stays small and stable; the crowd arrives when there's nothing left to reshuffle.

Upgrades, PME, and the unglamorous residue

Underneath both improvements sat a long-overdue platform chore I owned end to end: upgrading Apache Ignite from the 2.3 line to 2.15. Old versions carried real costs — we were hitting partition-map exchange (PME) delays and cluster instability during topology changes, exactly the operations a restart is made of. Working through version compatibility, topology settings, and JVM configuration resolved the PME stalls and gave the startup changes stable ground to stand on. None of this makes a good demo. All of it shows up in the final number.

Results, honestly stated

Startup went from over 90 minutes to about 7 — roughly a 13× faster recovery path, and a >90% cut in the window where an intraday incident leaves the business exposed. I'll resist attributing the whole win to any single change: the parallel Cassandra load was the biggest lever, but cache pinning, staged joins, the Ignite upgrade, and PME fixes all contributed, and they compound — stable topology is what lets a fast load stay fast.

What transfers

  • Read the internals of what you already run. The token ring was sitting in the documentation the whole time, holding a free parallelism scheme.
  • Startup is a distributed protocol — design it like one. Who joins when is a correctness decision, not an ops detail.
  • Data placement beats data movement. The cheapest rebalance is the one that never triggers.
  • Optimization moves bottlenecks; it rarely deletes them. Parallelizing the load shifted the pressure from one slow loader onto Cassandra's request queues — which is exactly where we wanted it, and we tuned the cluster to absorb it. When you speed something up, find where the strain lands next and decide, on purpose, that it's the right place for it.
  • Stability work multiplies speed work. The upgrade and PME fixes bought no headline number of their own; they made the headline number repeatable.