- What You Will Learn
- The Challenge of Big Data
- Today's Big Data Explosion
- Background for This Book
- Why the Focus on Database Sharding?
- Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
Background for This Book
My favorite subject is databases—pretty much everything about them, from database performance to management and of course scalability most of all. You might say I’m a bit of fanatic about it (okay, maybe I’m an extreme fanatic), but I wouldn’t do it if I didn’t love the field so much. I have been working intensively with all types of database technologies throughout my career. In one of my first companies, we built a successful independent database and application development firm, beginning with Sybase and expanding into other DBMS engines after that. More recently I ran Rogue Wave Software, a C++ tools vendor with a large concentration of Wall Street customers. While working with these customers, which included some of the top architects in the world, I saw a clear trend that spurred my interest even further: No matter how well an application could be scaled, the database tier was always the bottleneck. In fact, I learned that the database tier forms the last mile of application scalability, being at the very foundation of all functionality. The message was clear: Find a solution to that problem, and the potential was huge.
I have experienced the very best (and the very, very worst) that database management systems have to offer, in the trenches and firsthand. I’ve had the fortune and privilege of serving as the architect and manager for many mission-critical applications, from the advent of client-server technology to the Big Data systems of today. In addition to having lived through just about every database nightmare you can imagine, I’ve also seen the results when the database tier is performing outrageously well. I’ve seen how hard great software teams work to conquer these problems; there are few careers that require as much intelligence, persistence, and long, demanding hours. This has given me a profound respect for sharp technologists that conquer Big Data problems, the engines that are the very backbone of every application, not to mention the foundation of much of today’s economic growth. In fact, it’s obvious that the database explosion of the information age is upon us, and it’s likely to continue for many years to come as new and clever business opportunities arise to capitalize on this first-ever phenomenon encountered in world history.
Through this extensive and often intense experience, I’ve gained many insights over the years. My intent in writing this book is to share this experience with a wide audience, with the hope that you can avoid many of the common pitfalls I have seen repeated over and over. The book is written from a practical point of view, with what I believe are the salient points required to make the database tier of an application really cook. Many of the techniques can be applied easily, without the need to purchase or implement new technology, while others are intended as guides for taking full advantage of the latest and greatest database capabilities emerging from the fantastic technical advances made available in the past few years.
As CEO and CTO of CodeFutures (the makers of dbShards and AgilData technology), I have had the pleasure of working with the sharpest technical teams in my career—not just within our own engineering team, but also the customers that demanded the absolute best in database performance and reliability. We have worked with some of the world’s largest social networking applications, with phenomenal data growth and transaction volumes. Frankly, in the early days of our technology, we lived through a few genuinely hair-raising experiences, but with hard work in partnership with our customers, we always were able to succeed, even against the toughest challenges. More important, the wide array of applications where we have applied database sharding resulted in a broad range of experience, with large data sets and transaction volumes that were unimaginable just a few years ago. Many of these applications experienced extreme growth in transaction volume, an extraordinary thing to behold (to put it mildly). For example, watching a social game grow from nothing to over 1 million daily users in a few short months is a sight to see, and when you are part of the team responsible for ensuring the database platform handles the load, continuously running 24X7, it’s an experience that really grabs your attention (and can result in some very late nights!).
As a disclaimer right up front, many of the examples and experiences are derived from our work with our own products. dbShards, and our newest offering, AgilData, are most definitely commercial products, and of course part of my purpose in writing this book is to gain more attention for what we deliver. Having said that, I have worked hard to keep the book balanced, concentrating on the important facts that drive database scalability irrespective of a particular DBMS engine or technology. In fact, the book discloses a lot of the “tricks” and designs we use to deliver such scalable performance, with the hope that you will find them beneficial and useful in your own database environment. And don’t worry, we have new capabilities coming out with our advanced AgilData technology that will blow the world away. Who knows, they may provide material for future books in this series.