Behind the Curtain: Database Whoa’s

Aside from the newly released Custom Response Buttons, we did a bit of work behind the scenes relocating our main database server to a beefier piece of hardware in our Texas datacenter.
This resulted in around 0.1s of increased lag when loading up a page through our website or mobile interfaces, barely noticeable in most cases.

Our support team however started noticing huge slowdowns in their ability to browse through our websites, along with a couple of administrators for large counties with multiple agencies.
It turns out one of our authentication methods is using multiple queries for each agency a user is a part of, which doesn’t pose much of a problem when turnaround to the database is 10ms, but quickly adds up when queries start taking 100ms or so.

I’ll be working on optimizing this, either by removing the extra queries and putting them into the workflow somewhere else, or by combining them into an already existing query.

The advantage of putting the queries elsewhere in the workflow are that they will no longer affect every page request, and will only be called when needed.
The disadvantage of course is the proliferation of operations in our interface, which is rapidly growing as we add more nuances and functionality. We have talked about a web interface revamp in the near future, so this may be the correct solution until we can get to that project.

The advantage of putting the queries into one big one is that it will keep the logic in the same place, thus keeping our workflows streamlined and simple.
The disadvantage is that the query may become so big an cumbersome that it is too complex to work efficiently, as well as maintain in the future.

Either way, we are due for a reworking of the website in the near future as neither is a great solution, and will require some restructuring in order to remain scale-able for the future.

Behind the Curtain: Holiday Hacking

I had finished the iOS and Android pieces of our custom response buttons before driving off to our annual holiday stay at my in-laws, but had completely forgotten Webview and SMS.

The custom response buttons was on top of your wishlists, and we wanted to be sure nobody felt like the only kid without a Red Ryder BB Gun.

The advantage of working with 1’s and 0’s is that you can do it pretty much anywhere. After sneaking in some hacking between bites of holiday ham and unwrapping presents, I was able to push out the final pieces for response buttons today to all our agencies.

Happy Holidays everyone, don’t shoot your eye out!

Big(ish) data and loving it

I was sitting here, in the dark, at 2 AM and wishing that Facebook would allow me to post under my real name. ?I work at Active911 but when I post on our Facebook wall, it all looks like the same person posting (you can never tell who “I” is).

The reason I wanted to use my real persona is because my feelings tonight are very personal. ?I just finished converting some 40 million or so records from one data type to another, so that Jonathan can proceed with his feature rollout (the much requested response button renaming). After I converted the records, I started working on our new sharded?data store (more on that some other day) and created a new Github repo called “bard” (rhymes with “shard” 🙂 ) to manage all of this. ?I’m currently planning on storing Thrift-serialized data in a MySQL blob column.

AND I FREAKING LOVE THIS STUFF. ?I guess it sounds boring to non techies, but this kind of stuff makes my engine purr. I love the stuffing out of my job.

Let’s hear it for everyone out there who loves what they do!

Behind The Curtain: The Bleeding Edge

We’ve often discussed the increasingly slow pace of deployments here at the office (not to be confused with The Office, our mistakes entail a lot less laughing and a lot more crying).
Unfortunately, as we grow larger, more complex, and with more agencies relying on our service, it becomes riskier to deploy quick late-night hacks at a moment’s notice.
We usually go through at least 2 weeks of testing, and even then we can’t anticipate all the corner cases that are out in the field.

We’ve toyed with the idea of bringing up a secondary system that would be deployed to much more frequently, but with things breaking frequently as well.
The benefit for agencies would be more frequent updates, and for us would be finding bugs faster.
The downside of course is that it could, and would, break at any moment, though the fixes would also be deployed faster as well.

Would anyone be interested in signing up for this sort of bleeding edge system for their agency, or would the instability make it pretty much unusable to anyone?

Behind the Curtain: OpenAPI and OAuth2.0

As we progress in developing our OpenAPI, we’ve come across many use cases that are similar to what lots of people have already done, ie giving access to protected resources to a 3rd party app.
As such, we’ve decided to scrap our homegrown authentication scheme and go with doing OAuth2.0 style authentication instead.
This will allow users to enable and revoke access to the application on a per-user basis rather than the application as a whole.
If anyone has any experience with OAuth2.0 or any other authentication architecture, such as OpenID, and would like to share their experience, we are not set in stone on this and still have a ways to go in terms of development. Please feel free to comment below on your experience.

Behind the Curtain: Thrifty Development

As we move to systems that scale even better and faster than our current servers, we have begun to move towards using Apache Thrift as our communication layer more and more. ?Today I’m tinkering with it to try and build out our existing API, replacing with a more RESTful interface.

As a somewhat less-used architecture, there’s not a lot out there about Thrift. ?In particular, not very much about implementing their PHP Client is available out there, even the wiki page for PHP examples is conspicuously empty. ?There is even less in regards to the latest 0.9.0 release, which is already over a year old at this point.

There are a couple of tutorials. ?One of the main ones being referenced is?http://chanian.com/2010/05/13/thrift-tutorial-a-php-client/, which is already 3 years old. ?It uses 0.2.0, which has a vastly different directory structure than the current one. ?Still, it allowed me to figure out at least that the library files are all housed in the lib folder of the Thrift tarball.

The second tutorial I came across was?http://sleeplesscodingnights.blogspot.com/2012/11/installing-thrift-090-for-java-server.html, which does deal with Thrift 0.9.0 and implementing a Java Server with a PHP Client. ?However, at Step 7, I found myself getting frustrated with the many files that were required in the Thrift generated classes.

My particular classes required a few more than the ones specified in the tutorial. ?Over and over the php interpreter would fail on one class, and I would diligently find it in the lib/php/lib/Thrift/ folder and add another require_once statement. ?After about 5 such classes, I decided there must be a better way and started poking around the tarball to see what else I could find that could guide me in making my simple RESTful client work.

I find myself digging into the tutorials under tutorial/php to glean what I can about the way in which the dependencies must be loaded, and have found that Thrift PHP now uses a ClassLoader to load in the classes, as shown by the php example included with the 0.9.0 release.

//This line is the path to the ClassLoader.
//Replace it with the full path to wherever you put the php libraries
require_once __DIR__.'/../../lib/php/lib/Thrift/ClassLoader/ThriftClassLoader.php';
use Thrift\ClassLoader\ThriftClassLoader;
//This is the path to your generated php files,
    which were originally put in a gen-php folder.
//Replace it with the full path to wherever you put the generated files
$GEN_DIR = realpath(dirname(__FILE__).'/..').'/gen-php';
$loader = new ThriftClassLoader();
//This registers the base 'Thrift' namespace of all the Thrift library files
//Replace the second parameter with the full path to the parent folder of your Thrift library folders
$loader->registerNamespace('Thrift', __DIR__ . '/../../lib/php/lib');
//This is unnecessary for your project, 
    it is a tutorial folder of generated classes under the 'shared' namespace
$loader->registerDefinition('shared', $GEN_DIR);
//This is the way in which your generated files are loaded.
//Replace 'tutorial' with your namespace
$loader->registerDefinition('tutorial', $GEN_DIR);
$loader->register();

Once that was completed, all I had to change was the Protocol since Thrift’s default Server implementations use TFramedTransport and the Thrift example uses TBufferedTransport.

Overall, this took about 3 hours to finally completely piece together. ?Hopefully this will save people a bit of time in implementing their own PHP Thrift Client.

If you want to try out some Thrifty development of your own, our Mule Server project may be a good starting point. ?It incorporates a fast in memory cache with a Thrift Service, and the generic example is for quickly retrieving and caching authentication requests. ?https://github.com/active911/mule-server

Behind the Curtain: Introduction

Hello everyone!

As I sit here trapped by 3 terrifying inches of snow (yes, coming to the Pacfic Northwest from New England, it’s comical to me too), I feel as if it is a good time to introduce myself to all of you out there.

Though some of you may know me as the person responsible for the Android app, I am sure many of you didn’t know until this moment that I even existed. ?Well I assure you, I do exist, and I break things just as often (if not more) than the rest of the team.

I joined Active911 early this past July, after several years working for Hewlett-Packard and Polycom. ?While I enjoyed and learned much from my tenure at these multi-national corporations, I often found it difficult to really feel that I was making a big impact on our customers.

At Active911, I know that every decision, big or small, directly affects each and every one of you. ?Though I don’t often personally post on Facebook or the forums, I do look at them at least twice a week to get a feel for what is working well and what is not.

I hope to use this blog as a way to give a little more insight into what is currently in development, especially things that are not as obvious as a new Android release or a new tab in the interface.

Currently, my goal for this month is to finish implementing the customizable response buttons for our Android and iOS apps, as well as add more functionality to our Open API. ?We are still compiling use cases for our API, so please let me know in the comments if there is anything particular you want to see available in it!