Squashing Bugs in the Android App

We made some pretty major changes under the hood in Android 1.5.7.5, which resulted in a few more bugs than usual. This week we have released 2 minor updates to fix some of the most reported bugs.

The app was crashing when:

1)  trying to load responders’ info into the map view,

2) opening the alert report, backgrounding the app, and then reopening the app in quick succession,

3) the background location update restarted after other apps used up all the resources on the device.

These crashes have been fixed, as well as, the permissions bug. We also received feedback about wanting the responders view sorting feature to stay sorted across all alerts. We have made that improvement with the latest version. You can find the update in the Play Store on your Android device.

Thank you for all your feedback and stay safe!

Webview Update

For some time now, a bug has been reported on Webview where it will not automatically reconnect without refreshing the page. It has been very difficult to figure out where the problem lies, whether it’s the devices internet connection or a problem with webview connecting to our servers. Being unable to successfully reproduce the issues on our machines, we needed to take a shotgun approach to at least narrow down the problem. Today we rolled some changes that will detect when webview cannot stay successfully reconnected and automatically reload the page for you. We believe this should fix the problem for most of our webview devices that disconnect sporadically, but have a relatively stable connection. For webview devices with an erratic and unstable connection, a warning will populate after reloading 20 times in a 48 hour period, indicating that webview’s has lost its internet connection at least 20 times in the past 2 days. If you see this warning, please follow the instructions shown on the error and, if you require additional assistance diagnosing internet connectivity, contact us at support@active911.com.

Backend Update – Bugfix: The app taking a long time to load recent alerts

We made a change to the access server this week that will help people who have multiple agencies with the same device code. There was a bug that could introduce long load times for alerts when you opened the app after a certain period of time.

Basically, whenever our app is opened it calls our access server for all of the unread alerts for that device, then attempts to download the alerts that have been sent while the app was closed. When devices did this, they did it for all the agencies they had ever been a part of. If you were part of a department a year ago, we were still trying to get all the messages sent to that department since you had left. Depending on how busy that department was, it could add anything from seconds to minutes to the message downloading process.

To fix this issue, the initial call for what alerts your device needs to download longer try to get ones that belong to agencies you are no longer a part of. We still keep those alerts in the database, so if you accidentally remove yourself from an agency and then add yourself back, your device will be able to fetch those unread alerts.

It took us a long time to recognize the problem, but we fixed it as soon as we figured it out. For everyone impacted by this bug, we thank you for your patience and apologize for not finding it sooner.

Android Version 1.5.7.5

Features

Background Location

This main feature for this release is the Background Location reporting feature.

It works in much the same way as the iOS feature,  but a key difference is that Android has no indication when apps are using background GPS.

To turn the setting on, go to your setting in the app, and select the “Background GPS Mode” setting.  This will show you the 3 options for the Background GPS.

Screenshot_20170704-160537  Screenshot_20170704-154304

The top option, “High Accuracy”, will update the location of the device in the background at the same frequency it would while the app is opened.  This will likely increase battery usage by a fair amount, but will give the most accurate position and update the most frequently.

The middle option, “Battery Saving”, will use update the location of the device  at most once per 5 seconds. This will use less battery, but also update less frequently.

The bottom option is “Off”, which is selected by default, means the app will not report your position after you have backgrounded it.

Important Notes:

This new setting only applies when Active911 is in the background, it has no effect on the position reporting when the App is open, the Foreground GPS setting still controls that.

If you reboot your device, the Active911 app must be opened at least once before Background GPS will start working again.

Also, if you are ever stationary for more than a few minutes, the app will stop reporting your position until you start moving again. Therefore, you may disappear from the map, but then reappear when you start moving again.

 

Drag and Drop Map Markers

Long-pressing on a Map Marker will now allow you to drag the marker to a new location on the map.

Screenshot_20170704-154556

You can drag multiple markers, and then press “Save” to save the location of all the markers.

Important Notes:

If you drag multiple markers and press “Edit”, the edit screen will still be for the first marker you started dragging.  The same holds true for the “Delete” button.

This Drag and Drop control, as well as editing map markers, is being redesigned in an upcoming Android release to make it easier to use.

Sorting Responders View

In the responders view for an alert, you can now tap the button at the top of the view to toggle between sorting by the device name and sorting by the response type.

Screenshot_20170704-154354

Location Following

The last change made to the Android app is the way our “Follow My Location” feature works on the map.

Screenshot_20170704-161821

Previously, whenever you long-pressed on the “My Location” button on the map, it would pan and zoom to your location at a default zoom level, then pan to keep you location centered on the map.

This caused some frustration for users who wanted to be zoomed in to a specific level while having it follow them.

We’ve now changed it so that, upon entering “Follow My Location” mode, the map only pans to your location without zooming.

We know that some people do like the zooming functionality of the button, so we have left single-tapping on the “My Location” button the same.

To zoom to your location and have it follow you, you just need to single tap on the “My Location” button, wait a second for it to zoom to you, and then long-press on the button.

Behind the Curtain: New Icons

nfpa170icons

 

New icons are something that have been requested from numerous agencies over the past couple of years.

In order to provide icons that will be useful to everyone, we started looking at some iconography standards in the fire industry.

One that stood out as being widely adopted with a large variety of icons is the set defined by the NFPA 170 standard.

We’ve begun the process of adding these icons to our system, and they will be available in the near future to all agencies.

Aside from allowing a wider variety of icons, we hope that this will foster more standards based iconography on map data, allowing agencies to be more inter-operable going forward.

If you have any other standards based iconography that your agency uses, please let us know so that we can evaluate them and possibly add them to our system.

New Web Console Features

We’ve added a few new features to the web console last week.

  • We’ve added a “Bulk Edit” button under the map data tab which allows you to change all map markers of one type/color to another type/color.  It also allows bulk deletion of the map data.
  • You can now specify a different device name for each agency the device is a part of.  Thus, when you change the name of a device in one agency, it will not affect its name in other agencies.
  • There is a new button next to each pagegroup which allows linking to another pagegroup.  This will forward all alerts sent to the pagegroup to all of its linked pagegroups.

Let us know how these features work for you!

Behind the Curtain: The Future is Now

Over the past couple of years, we’ve worked hard to release the latest and greatest new features as quickly as we can.

As time has gone on, and the system has grown larger and larger, being able to roll out new features has quickly become a much more lengthy process.

We’ve repeatedly told ourselves that, in the future, things will be better, but for now we need to keep rolling on with the system the way it is.

Well, after our lengthy Assignments projects, we’ve come to realization that the future has finally arrived, and we need to make the radical shift we’ve been pushing off for too long.

The main thing we are planning for the rest of this year is a complete Beta Network: A complete network of all our servers which will act as a bleeding edge system.  For agencies which want the latest and greatest, the beta network will have continuous integration with code as we produce it.  The beta network would have less stability and reliability, so users who can wait for our scheduled releases would remain on the main production services.

In conjunction with this, we are looking at developing a better system of community feedback, and identifying users and agencies which can help us vet our systems thoroughly before they go live to production.

If you have any ideas on the best ways we can achieve these goals, we are always welcome to suggestions from the community, just leave a comment below.

We will also be doing another Live AMA today (July 31) at 2PM Pacific time, so click here to submit any questions about any of our products or the direction we are going in for the rest of this year

 

Behind the Curtain: Clustering Across The World

In my last post, I discussed the obstacles introduced by restructuring our services into a more modular and scalable structure.

One obstacle this has introduced is how even a small amount of latency between the backend and interfaces could dramatically increase the time it takes to fulfill a complex request.
This becomes a very real possibility when considering that we are planning to distribute our services into multiple datacenters across the world in order to provide constant uptime and availability, even in cases of local natural disaster.

To do this, our shards are provisioned in such a way as to group geographically similar objects together on the same shard. This means that, for the most part, a single agency, as well as agencies which are close together, will likely be on the same shard.

By incorporating these shard identifiers into the new 64-bit identifiers themselves, we are able to direct requests to a particular shard cluster just by using a DNS prefix for each shard identifiers.
This allows us to integrate our services vertically within each cluster, from the frontend apis all the way down to the database backing each shard.

There will be some objects which may be sharded to a location geographically distant from their original agency, for example when someone moves across the country and switches to an agency in their new city. Thus, there will be some connections between distant clusters, but we expect this will be far and few between.

In essence, each cluster will be able to serve a distinct geographic area with minimal connections to other shards, allowing consistently low latency between the layers of our backend infrastructure.

We will be migrating data to the new system slowly over the course of the next year, which should create minimal impact on our current system. This concludes our preliminary look at our new sharded data infrastructure, please join me next time as I present an overview of where Active911 is today and what we hope to achieve through the end of 2015 and into 2016.

Behind the Curtain: Sharding Over Time

I’ll explore my first topic to do with sharding in a bit more depth today.

The first obstacle to tackle has been increasing the number of requests needed to fulfill some of our API calls.
For example, whenever a device calls home for an update, it asks for all of the agencies and devices which are related to it, as well as all of the alerts that the device hasn’t read yet.
In our current single database structure, this is basically 3 database calls, albeit fairly heavy ones.
Within a sharded system however, a call must be made for each individual related object since each could live on an entirely different server located hundreds of miles from the original device.

Now, the heavy complex query may each take 140ms to establish a connection, 200-400ms to run each query, and 100ms to transmit the response.
Making 3 such calls equates to about 2 seconds.
Comparatively, in a sharded system, a database call would need to be made, in the worst case, once for each related device and agency.
The average device belongs to about 3 agencies, each agency contains on average 25 devices.
That means a single call home would require at least 80 database calls.
In a naive implementation, that would be 140ms for each database connection, 5ms to run each query, and 5ms to transmit each response.
As you can see, this adds up to about 12 seconds for a single call home, a drastic increase.

Of course, there are methods by which we can reduce this. For example, taking advantage of the fact that, since most people will be part of geographically similar agencies, the likelihood of them being all on the same server cluster is high. This allows us to implement things like connection pooling and multiple select queries when we are able to anticipate records being on the same shard server.
Connection pooling alone would likely decrease the above example from around 12 seconds to about 2.4 seconds, already bringing it closer in line with our current performance.

The one thing that makes all of this even more complicated is the fact that, the more we shard things, the more prone to latency issues it will be. A 25ms delay between the interface and the database in our current system equates to about 75ms longer api calls, a decent but mostly unnoticeable difference.
In the world of shards, this would translate to a whopping 6 second increase, almost quadrupling the average use case.
That leads us to the next obstacle, latency caused by widespread geographic distribution of our services, which we will tackle in my next blog post about clustering of services.

Behind the Curtain: Scalability

As Paul noted in his post, we’ve been working on building up our backend to be more scalable, especially our Databases. As it currently stands, we have a single database which is loaded into memory on startup. This allows all of our queries to be super fast, event when traversing hundreds of thousands of rows.

There are a few downsides to our current deployment. First, since we’re preloading all the data into memory, the spool up time for this database is huge. Since the cost is so high, taking it down for maintenance is something we do very infrequently. While this is not a bad thing in and of itself, it does mean that we cannot make large changes to our database structure more than a couple of times a year.

Second, since all the data is on a single server, the only way to scale up is to throw more resources at the server. This is only feasible as long as our growth is at or below the pace which technological power increases. As we grow faster and faster, this is becoming less and less feasible, necessitating the need to more toward a more distributed datastore system.

Third, serving all of our data from a single datacenter means that connectivity issues in that area would adversely affect service nationwide.

Sharding solves all 3 of these issues.

First, since each shard contains just a fraction of the data, it takes much less time to warm up each server. It also allows us affect only a small set of our users if we need to take a server down for maintenance.

Second, sharding allows us to gradually bring up more servers as needed to serve the growing load. This is especially useful in allowing high volume areas to have their own dedicated cluster of shards.

Third, by splitting the data up across multiple shards, we can begin widely distributing our servers. We have long discussed and wanted to implement a way for us to host the servers away from the areas they service. For example, New York would be serviced mainly by servers hosted outside of the Northeast. Not only would this resolve the single point of failure issue that Paul outlined, it would also decrease the likelihood that our service would go down during a natural disaster, since the servers are outside of the affected area.

Of course, such a huge change will take time, and is not without its risks and potential downsides. We are making this change gradually, starting with the new PARS board feature that is coming soon. Check back often as we explore the wonder and mysteries of sharding.