Turning the database inside out

As our industry progresses through its cloud-native metamorphosis, we are also experiencing a revolution in database thinking. The amount of advancement we have seen in database technology over the past decade is significant. It started with the NoSQL movement, but then expanded into the Polyglot Persistence arena. Database sharding techniques finally enabled databases to scale horizontally by spreading disk access across commodity hardware. Polyglot Persistence expanded on this technique by optimizing how data is stored on these disks for different read and write patterns. CAP Theorem taught us to embrace eventual consistency in our designs in favor of availability.

However, with this advancement comes a significant increase in operational complexity and the workforce needed to administer these databases. With this overhead comes the tendency to operate in a shared database model. As was previously mentioned, this means that there are no bulkheads between the various components sharing these monolithic database clusters. Without isolation, at best these components just compete for the cluster's scarce resources and impact each other's performance, at worst a catastrophic failure in the database cluster causes an outage across all the components.

Yet, in most cases, a company's value proposition is its data. A company's most valuable asset warrants more than a shared database model. This is where the real revolution is happening; a complete break from monolithic thinking at the persistence layer. We are turning the database completely inside out, replicating the data across many databases of different types to create the necessary isolation, outsourcing some of the responsibilities of these databases to the cloud provider, and leveraging the innovations of cloud-native databases, such as global replication. Let's dissect how we turn the database inside out.

We need to understand a bit about how databases work, to understand how we are turning them inside out. Martin provides a very thorough description in his article, which I will paraphrase here. At the heart of any database is the transaction log. The transaction log is an append-only structure that records all the events that change the state of the data. In other words, it records all the insert, update, and delete statements that are executed against the database. The database tables hold the current state representation of these events. The transaction log can be replayed to recreate the current state of the tables if need be. This is what happens during replication. The logs are replayed against another node to bring it up to date. The database also manages indexes. These indexes are just another copy of the data sorted in the desired sequence.

Traditional databases are notoriously slow and contention for connections is high. This drives the need for caching query results on the consuming side. A cache is yet another copy of the data, but this copy can be incomplete and stale. Performance can be very slow when we have complex queries over large sets of data. To solve this problem databases maintain materialized views. A materialized view is yet another copy of the data, which is the result of a query that is continuously kept up to date by the database. These are examples of derived data, that, other than the cache, are managed by a single database cluster. We ask our databases to do a great deal of work and then wonder why they are slow.

Our objective is to take all this processing and turn it inside out, so that we can spread this processing across the cloud, to achieve massive scale and sufficient isolation. Lets first revisit the Reactive properties. Our components should be responsive, resilient, elastic, and message-driven. Our message-driven inter-component communication mechanism is event streaming. As depicted in the preceding diagram, the event stream is the externalized transaction log. Components publish their events to the stream. Other components consume these events from the stream and create read replicas of the data in their own Polyglot Persistence stores by programmatically populating materialized views (that is, tables) that are optimized for their own usage. These local representations of the data help the system achieve Reactive properties.

First, they make a component responsive by providing an optimized cache that effectively eliminates the stale cache and cache miss problems, because it is continuously warmed. Second, they act as a bulkhead to make a component resilient to failures in upstream components, because the component will always retrieve the latest known state from its own storage. Finally, this makes the system extremely elastic, because the read replicas are spread across many databases in many components. The main patterns at work here are the Event Sourcing and CQRS patterns. We will discuss these in Chapter 3, Foundation Patterns, and Chapter 4, Boundary Patterns, respectively.