Scaling GIS Data in Non-relational Data Stores

Editor's Notes

• #4: So first of all, I’ve been head-down coding like 14 hours a day for the past couple weeks. So these slides aren’t as polished as I’d like them to be. Luckily, I’ve been working on exactly this stuff, so it’s all in the front of my mind. And since this is a workshop I can be more interactive. Interrupt with questions any time!
• #6: I’ve been interested in GIS for a while, but I’m relatively new to the scene. I’ve done lots of work building scalable websites though. And, honestly, that’s a problem that’s more or less solved.
• #7: HTTP has no “session state,” but applications that communicate via HTTP do have to maintain state. Without application state there’d really be no reason to have a web site or web service - if there’s no application state (nothing the web server knows that the client doesn’t) then the algorithm can be completely distributed.
• #9: And, the truth is, relational databases are pretty awesome. They have a number of great characteristics. They’re well understood. And they’re robust.
• #11: Most RDBMS systems are ACID compliant or at least pretty close. These characteristics allow client code to make simplifying assumptions about the data that is returned from the data store.
• #12: RDBMS weaknesses are essentially the inverse of their strengths.
• #13: The upshot of all this is that write performance is much poorer than read performance. But writes are much harder to scale than reads because they have to happen on an authoritative node. Reads can be scaled easily using replication.
• #15: Popularized by Eric Brewer at Principles of Distributed Computing in 2000. Brewer’s Conjecture. Later formally proven. You can design a scalable architecture that maintains some of the ACID characteristics of a typical relational datastore, but eventually you’ll have to relax some of these constraints. Note that consistency, as defined here, does not mean the same thing as “consistency” in “ACID” - before it meant that all data constraints were met.
• #16: So this is probably something you’ve heard before. And people often just throw it out there without an explanation. But it’s pretty easy to prove to yourself by contraction, so let’s try that.
• #17: Node A and B are a master/master pair, so they replicate data to one another. To meet the ACID requirements both nodes have to write the data durably before one of them can respond successfully to a write.
• #22: Abstraction layers: sharding, caching, and client-side replication are basically kludges on top of relational data stores that trade consistency for availability and partition tolerance - but why re-invent the wheel?
• #23: Specialized data stores: graph databases, document databases, key-value stores, and various combinations.
• #24: Facebook’s Hive project adds SQL on top of HBase, but SQL queries are translated into map-reduce jobs that are run across the distributed system
• #25: Specialized data stores: graph databases, document databases, key-value stores, and various combinations.
• #26: We should probably call these datastores NoACID instead of Nonrelational or NoSQL.
• #27: That’d make much more sense. But I digress.
• #28: Eventual consistency is a concept that was popularized by Amazon CTO Werner Vogel. This is a gross simplification, and the approaches data stores take to perform node recovery, rebalancing, and repair are often their most distinguishing characteristics. This is actually why we chose Cassandra - the distributed cluster logic is more robust than any other store I’ve seen.
• #32: The outermost layer is the key.
• #33: Column families are stored together on disk and are the next layer of the structure.
• #34: And finally we have columns. You can have as many of them as you want, and each row and column family can have different columns. It’s schema-less - thus non-relational.
• #36: Cassandra partitioners are used to decide which node data should be stored on and which node responds to a query.
• #37: If we can get our data to fit model where we’re simply retrieving items by key from a sorted set then it’s pretty easy to store and query efficiently. Anything more complicated usually requires heuristics and deep insight into the data set to do at scale.
• #38: At the very least we need to index on (latitude, longitude)... may also need to through altitude and time into the mix. That’s four dimensions.
• #39: Space-filling curve: developed by Peano, refined by Hilbert.
• #43: The non-algorithmic approach is a massive undertaking that requires constant attention and involves a large amount of ambiguity.
• #50: Ask me questions!