This chapter is from the book
This chapter is from the book
This chapter is from the book
Deploying Spark on Hadoop
As discussed previously, deploying Spark with Hadoop is a popular option for many users because Spark can read from and write to the data in Hadoop (in HDFS) and can leverage Hadoop’s process scheduling subsystem, YARN.
Using a Management Console or Interface
If you are using a commercial distribution of Hadoop such as Cloudera or Hortonworks, you can often deploy Spark using the management console provided with each respective platform: for example, Cloudera Manager for Cloudera or Ambari for Hortonworks.
If you are using the management facilities of a commercial distribution, the version of Spark deployed may lag the latest stable Apache release because Hadoop vendors typically update their software stacks with their respective major and minor release schedules.
Installing Manually
Installing Spark on a YARN cluster manually (that is, not using a management interface such as Cloudera Manager or Ambari) is quite straightforward to do.
Try it Yourself: Installing Spark on Hadoop Manually
Follow the steps outlined for your target platform (for example, Red Hat Linux, Windows, and so on) in the earlier section “Installing Spark in Standalone Mode.”
Ensure that the system you are installing on is a Hadoop client with configuration files pointing to a Hadoop cluster. You can do this as shown:
hadoop fs -ls
This lists the contents of your user directory in HDFS. You could instead use the path in HDFS where your input data resides, such as
hadoop fs -ls /path/to/my/data
If you see an error such as hadoop: command not found, you need to make sure a correctly configured Hadoop client is installed on the system before continuing.
Set either the HADOOP_CONF_DIR or YARN_CONF_DIR environment variable as shown:
export HADOOP_CONF_DIR=/etc/hadoop/conf # or export YARN_CONF_DIR=/etc/hadoop/conf
As with SPARK_HOME, these variables could be set using the .bashrc or similar profile script sourced automatically.
Execute the following command to test Spark on YARN:
spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster $SPARK_HOME/lib/spark-examples*.jar 10
If you have access to the YARN Resource Manager UI, you can see the Spark job running in YARN as shown in Figure 3.4:
FIGURE 3.4 The YARN ResourceManager UI showing the Spark application running.
Clicking the ApplicationsMaster link in the ResourceManager UI will redirect you to the Spark UI for the application:
FIGURE 3.5 The Spark UI.
)
)
Submitting Spark applications using YARN can be done in two submission modes: yarn-cluster or yarn-client.
Using the yarn-cluster option, the Spark Driver and Spark Context, ApplicationsMaster, and all executors run on YARN NodeManagers. These are all concepts we will explore in detail in Hour 4, “Understanding the Spark Runtime Architecture.” The yarn-cluster submission mode is intended for production or non interactive/batch Spark applications. You cannot use yarn-cluster as an option for any of the interactive Spark shells. For instance, running the following command:
spark-shell --master yarn-cluster
will result in this error:
Error: Cluster deploy mode is not applicable to Spark shells.
Using the yarn-client option, the Spark Driver runs on the client (the host where you ran the Spark application). All of the tasks and the ApplicationsMaster run on the YARN NodeManagers however unlike yarn-cluster mode, the Driver does not run on the ApplicationsMaster. The yarn-client submission mode is intended to run interactive applications such as pyspark or spark-shell.