- Introduction
- Apache Hadoop
- Phase 0: The Era of Ad Hoc Clusters
- Phase 1: Hadoop on Demand
- Phase 2: Dawn of the Shared Compute Clusters
- Phase 3: Emergence of YARN
- Conclusion
This chapter is from the book
This chapter is from the book
This chapter is from the book
Phase 0: The Era of Ad Hoc Clusters
Before the advent of ad hoc clusters, many of Hadoop’s earliest users would use Hadoop as if it were similar to a desktop application but running on a host of machines. They would manually bring up a cluster on a handful of nodes, load their data into the Hadoop Distributed File System (HDFS), obtain the result they were interested in by writing MapReduce jobs, and then tear down that cluster. This was partly because there wasn’t an urgent need for persistent data in Hadoop HDFS, and partly because there was no incentive for sharing common data sets and the results of the computations. As usage of these private clusters increased and Hadoop’s fault tolerance improved, persistent HDFS clusters came into being. Yahoo! Hadoop administrators would install and manage a shared HDFS instance, and load commonly used and interesting data sets into the shared cluster, attracting scientists interested in deriving insights from them. HDFS also acquired a POSIX-like permissions model for supporting multiuser environments, file and namespace quotas, and other features to improve its multitenant operation. Tracing the evolution of HDFS is in itself an interesting endeavor, but we will focus on the compute platform in the remainder of this chapter.
Once shared HDFS instances came into being, issues with the not-yet-shared compute instances came into sharp focus. Unlike with HDFS, simply setting up a shared MapReduce cluster for multiple users potentially from multiple organizations wasn’t a trivial step forward. Private compute cluster instances continued to thrive, but continuous sharing of the common underlying physical resources wasn’t ideal. To address some of the multitenancy issues with manually deploying and tearing down private clusters, Yahoo! developed and deployed a platform called Hadoop on Demand.