Behind the Curtain: Clustering Across The World

In my last post, I discussed the obstacles introduced by restructuring our services into a more modular and scalable structure.

One obstacle this has introduced is how even a small amount of latency between the backend and interfaces could dramatically increase the time it takes to fulfill a complex request.
This becomes a very real possibility when considering that we are planning to distribute our services into multiple datacenters across the world in order to provide constant uptime and availability, even in cases of local natural disaster.

To do this, our shards are provisioned in such a way as to group geographically similar objects together on the same shard. This means that, for the most part, a single agency, as well as agencies which are close together, will likely be on the same shard.

By incorporating these shard identifiers into the new 64-bit identifiers themselves, we are able to direct requests to a particular shard cluster just by using a DNS prefix for each shard identifiers.
This allows us to integrate our services vertically within each cluster, from the frontend apis all the way down to the database backing each shard.

There will be some objects which may be sharded to a location geographically distant from their original agency, for example when someone moves across the country and switches to an agency in their new city. Thus, there will be some connections between distant clusters, but we expect this will be far and few between.

In essence, each cluster will be able to serve a distinct geographic area with minimal connections to other shards, allowing consistently low latency between the layers of our backend infrastructure.

We will be migrating data to the new system slowly over the course of the next year, which should create minimal impact on our current system. This concludes our preliminary look at our new sharded data infrastructure, please join me next time as I present an overview of where Active911 is today and what we hope to achieve through the end of 2015 and into 2016.

Leave a Reply

Your email address will not be published. Required fields are marked *