Real-Time Market Data Pipeline

The raw-data vendor market charges five-figure monthly fees for tick data with contractual latency floors and no completeness guarantees. The three major retail feed providers we evaluated each had documented gaps: Polygon's crypto depth dropped during high-volatility periods, Refinitiv's API introduced 200ms to 400ms buffering on OHLCV bars, and none of the three carried full BSE/NSE symbology alongside US equities in the same normalized schema. We needed full historical depth across equities, crypto, and macro series, queryable fast enough to power live dashboards and backtests running simultaneously.

The hard part is not the volume. The hard part is that each exchange has its own wire protocol, its own reconnect semantics, its own rate-limit envelope, and its own definition of what a sequence number means. Binance sends delta snapshots and expects you to maintain an order book locally. OKX drops the connection silently every 30 seconds unless you send a ping in their proprietary format. Kraken uses a channel-subscription model that does not map cleanly onto a symbol-centric schema. Building a single Python process to handle all of this meant one exchange's timeout caused an async event-loop backlog that delayed downstream writes for every other exchange by 100ms to 600ms.

We tried a Python aiohttp fan-in approach first. It worked at three exchanges. At seven it began dropping ticks during volatile sessions because asyncio's default executor pool backed up on JSON parsing. We needed full per-exchange isolation with a shared output contract.

Each exchange connector runs as an independent Tokio async task compiled into a single Rust binary. Tasks share no state except a reference to a Redis client. The Rust async model gives us M:N scheduling across a fixed thread pool of 16 threads matching the CPU count of the host machine, so 10 concurrent WebSocket connections idle at under 0.5% CPU total between market sessions.

The data flow is a three-stage fan-in. Stage one: each connector task maintains its own WebSocket connection with heartbeat handling, sequence tracking, and exponential-backoff reconnect. On reconnect it queries a per-exchange snapshot endpoint to backfill any ticks missed during the gap, then resumes streaming. Stage two: each normalized tick is published to a Redis Stream keyed by exchange:symbol. We chose Redis Streams over Kafka because the operational overhead of a Kafka cluster for this throughput is not warranted; a single Redis 7.2 instance handles it with sub-1ms append latency. Stage three: a separate Rust persistence worker consumes the Redis Streams via consumer groups, batches writes into 10,000-row chunks, and bulk-inserts to ClickHouse.

The trickiest piece was OKX's silent disconnect behavior. Their WebSocket drops the connection exactly 30 seconds after the last server-to-client message if no ping was sent in the prior 25 seconds. The wrinkle: their server-to-client ping frame is not a WebSocket ping frame; it is a raw string literal sent as a text message. The tungstenite library's built-in ping/pong logic never fires because OKX does not use the protocol-level ping frame. We spent two days watching the OKX connector silently timeout before instrumenting per-message timestamps and realizing the 30-second wall. The fix was a concurrent heartbeat task spawned alongside the receive loop, sending the text string every 25 seconds.

Sequence tracking across Binance required maintaining a per-symbol last-seen update ID in Redis. If the invariant U == last_seen + 1 breaks, we have a gap. Gap recovery: discard the local book state, request the depth snapshot REST endpoint, then replay deltas from that point. This happened roughly 3 to 5 times per trading day per high-volume symbol during our initial deployment, mostly at the open.

Error handling follows a three-tier policy: transient network errors get exponential backoff up to 30 seconds before reconnect; exchange-side errors page on-call immediately; data integrity errors quarantine the offending row to an error stream for review rather than dropping silently.

Measured over a 14-day window ending May 1 2026, during US market hours as the high-water-mark period. We store 723,386,870 rows in bars_1m across 10,752 daily and 2,378 minute-resolution symbols. Ingest throughput peaks at roughly 80,000 ticks per second. ClickHouse write latency runs at 12ms p50 and 38ms p95. Redis Stream append latency holds at 0.4ms p50.

The 14.19 GB figure for 723M rows is the most important number. That is 19 bytes per row on average, achieved through Gorilla encoding on price columns plus ZSTD(3) at the block level. An equivalent Postgres TimescaleDB table at roughly 160 bytes per row uncompressed would require 115 GB for the same data.

The pipeline ran 14 days straight without a data-loss event. Forty-seven exchange-side disconnects were detected and recovered automatically. None resulted in a gap in the stored data because the snapshot-on-reconnect logic backfilled the missed interval before resuming.