- When Data Became a BIG Deal
- Data and the Single Server
- The Big Data Trade-Off
- Anatomy of a Big Data Pipeline
- The Ultimate Database
- Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
Data and the Single Server
Thanks to the constantly dropping price of commodity hardware, it’s possible to build larger and beefier computers to analyze data and provide the database backend for Web applications. However, as we’ve just seen, there is a limit to the amount of processing power that can be built into a single machine before reaching thresholds of considerable cost. More importantly, a single-machine paradigm provides other limitations that start to appear when data volume increases, such as cases in which there is a need for high availability and performance under heavy load or in which timely analysis is required.
By the late 1990s, Internet startups were starting to build some of the amazing, unprecedented Web applications that are easily taken for granted today: software that provides the ability to search the entire Internet, purchase any product from any seller anywhere in the world, or provide social networking services for anyone on the planet with access to the Internet. The massive scale of the World Wide Web, as well as the constantly accelerating growth of the number of total Internet users, presented an almost impossible task for software engineers: finding solutions that potentially could be scaled to the needs of every human being to collect, store, and process the world’s data.
Traditional data analysis software, such as spreadsheets and relational databases, as reliable and widespread as it had been, was generally designed to be used on a single machine. In order to build these systems to be able to scale to unprecedented size, computer scientists needed to build systems that could run on clusters of machines.