Latency-Aware Inference at the Edge
Discover how we route player sessions between Singapore, Virginia, and Frankfurt while keeping inference below 8ms — even during global peak hours.
AI-powered experiences are only magical when the model feels instant. The moment you can feel a pause between intent and response, the spell breaks. Now place that model on the other side of an ocean, add a few congested hops, and sprinkle in some jitter. That’s the reality for most players on global networks.
At Ludotronics, our goal was simple to state and hard to achieve: keep inference under 8ms for players anywhere on the planet, even when they’re deep in emergent, AI-heavy worlds. That forced us to think of inference not as a static endpoint, but as a dynamic, GEO-aware fabric that constantly reconfigures itself.
The tri-region backbone
Our backbone starts with three primary hubs: Singapore, Northern Virginia, and Frankfurt. They anchor the majority of our GPU capacity and give us good coverage for APAC, the Americas, and EMEA. Around them, we deploy smaller edge locations — from São Paulo to Sydney — that can host lighter models or cache warm activations.
Rather than hard-coding “players in Seoul go to Singapore,” we continuously measure effective latency and route based on live telemetry. Sometimes, congestion on a trans-Pacific cable makes Frankfurt a better choice for a player in Nairobi than Singapore, even if the geographic distance is larger.
Session-aware model placement
Inference routing is only half of the story. The other half is where the models actually live. For conversational agents, we can afford to cold-start weights in different regions. For dense world simulations with thousands of NPCs, that’s not an option.
We co-locate inference with the world graph shards that own a player’s current cell. When a squad moves from a shard in Oregon to one in Iowa, their associated models migrate with them. A background process continuously evaluates migration candidates by estimating cost: activation transfer size, expected session duration, and local GPU headroom.
Making latency the primary SLO
Most infrastructure SLOs start with uptime and throughput. We flipped the order: we treat latency as the primary product metric. It’s the number that shows up in design reviews and postmortems, not just in SRE dashboards.
Concretely, that means we allow certain non-critical features to degrade in favor of keeping inference snappy. If a region is under GPU pressure, we can temporarily down-sample background AI behaviors, reduce model size for non-player-facing agents, or skip certain analytics pipelines — all to protect the feeling of immediacy for the human on the other side.
Instrumentation at millisecond resolution
None of this works without visibility. Every inference hop is traced with millisecond resolution and enriched with GEO metadata: source ISP, country, approximate city, and even whether the player is on mobile or wired. That data feeds both our autoscaling decisions and our product roadmap.
For example, when we saw a spike of traffic from Istanbul consistently hitting our Frankfurt region with poor jitter characteristics, it pushed us to deploy additional capacity in nearby hubs and optimize our Turkish ISP peering strategy.
Latency-aware inference at the edge isn’t a one-time project; it’s a continuous negotiation between geography, hardware, and player expectations. The closer we push to the theoretical limits of the speed of light, the more the details matter — and we’ll keep sharing those details here as we iterate.