Six months ago, we hit a wall. Not the dramatic kind. No outage, no customer complaints, no spike in our error dashboard. Just a quiet, creeping realisation that the architecture we'd built to get SafiTrack from zero to a hundred customers was not going to carry us to a thousand.
The specific number that focused our attention: during peak hours on a Tuesday in November, we were receiving just over 800,000 location updates per day from devices in the field. We'd designed for a fraction of that. We needed to talk.
The original architecture and why it worked (until it didn't)
When we built the location ingestion pipeline, we made sensible early-stage decisions. A single Postgres instance handled everything: writes came in, we fan-out to the relevant dashboards via polling, done. Simple to reason about, easy to debug, fast to ship.
The cracks started showing at around 400K daily events. Write latency crept up. Dashboard refreshes felt sluggish. Our on-call rota started getting paged about database CPU at 2am. Classic growth pains, but ones we needed to address before they became customer-facing problems.
Step one: separate the write path from the read path
The first and most important change we made was pulling the write path entirely out of Postgres and putting it behind a Kafka topic. Devices now write to a lightweight ingestion service that validates the payload, stamps a server-side timestamp, and drops the event onto the queue. Postgres never touches a raw location update during ingestion.
This immediately took the pressure off our primary database. CPU dropped by around 40% during peak hours. More importantly, it decoupled ingestion from processing. We could now scale each independently.
Step two: consumer groups for different use cases
Once events are on the queue, we have three different consumer groups reading from them, each optimised for a different latency profile.
The real-time consumer processes events within about two seconds and writes to Redis. This is what powers the live map view. It doesn't need to be durable. If Redis loses a few seconds of state, the next update will correct it.
The analytics consumer batches events into five-minute windows and writes aggregated summaries to Postgres. This is what powers visit detection, route replay, and the intelligence layer. The write volume is 95% lower because we're writing summaries, not raw events.
The archive consumer writes the raw event stream to cold storage (we use S3) for compliance and historical replay. This runs on a thirty-second delay and we don't particularly care about its latency.
Step three: visit detection as a stream processor
Visit detection (the logic that says "this rep was at this location for long enough to count as a visit") used to run as a cron job every fifteen minutes. That was fine when we had a few thousand reps. At scale, it became a monster batch job that competed with everything else for database resources.
We rewrote visit detection as a stream processor sitting on top of the analytics consumer. As location events arrive, we maintain a stateful window per rep. When the conditions for a visit are met (proximity threshold plus dwell time), we emit a visit event. The database sees a small trickle of visit writes rather than a massive batch every quarter hour.
Step four: partitioning Postgres
Even with the reduced write volume, our Postgres tables were getting large enough that vacuum and autovacuum were causing noticeable performance degradation. We partitioned the visits and analytics tables by month. Old partitions get archived to a read-replica that the dashboard queries transparently. Hot partitions (last 90 days) stay on the primary.
Query performance on the dashboard improved dramatically. Not because we changed the queries, but because the planner was now only scanning a fraction of the data.
What we'd do differently
We should have introduced the Kafka layer earlier. We kept telling ourselves the simpler architecture was "good enough" until it wasn't, and the migration was harder than it needed to be because so many things depended on the original write path. The lesson is familiar but worth repeating: if you can see the shape of the scaling problem coming, it's almost always cheaper to address it before it's urgent.
We also underestimated how much operational complexity the streaming infrastructure would add. Kafka is powerful and reliable, but it's not free to run. We spent meaningful engineering time on monitoring, lag alerting, consumer group management, and handling the occasional partition rebalance at the worst possible moment. Plan for that cost.
The work isn't finished. We're already thinking about what 10M daily events looks like. But for now, the system is healthy, the on-call paging has stopped, and the team can focus on building product again rather than firefighting infrastructure. That feels like a win.