Managing and Understanding the Boot Procedure on Red Hat Enterprise Linux

This chapter is from the book

Red Hat RHCSA/RHCE 7 Cert Guide: Red Hat Enterprise Linux 7 (EX200 and EX300)

Learn More Buy

This chapter is from the book

This chapter is from the book 

Red Hat RHCSA/RHCE 7 Cert Guide: Red Hat Enterprise Linux 7 (EX200 and EX300)

Learn More Buy

Foundation Topics

Working with Systemd

Systemd is the new service in Red Hat Enterprise Linux 7 that is responsible for starting all kinds of things. Systemd goes way beyond starting services; other items are started from systemd as well. In this chapter, you learn how systemd is organized and what items are started from systemd.

Understanding Systemd

To describe it in a generic way, the systemd System and Service Manager is used to start stuff. The stuff is referred to as units. Units can be many things. One of the most important unit types is the service. Typically, services are processes that provide specific functionality and allow connections from external clients coming in. Apart from services, other unit types exist, such as sockets, mounts, and others. To display a list of available units, type systemctl -t help (see Listing 18.1).

Listing 18.1 Unit Types in Systemd

[root@server1 ~]# systemctl -t help
Available unit types:
service
socket
target
device
mount
automount
snapshot
timer
swap
path
slice
scope

Understanding Service Units

The major benefit of working with systemd, as compared to previous methods Red Hat used for managing services, is that it provides a uniform interface to start units. This interface is defined in the unit file. The system default unit files are in /usr/lib/systemd/system. System-specific modifications (overriding the defaults) are in /etc/systemd/system. Also, the runtime configuration that is generated automatically is stored in /run/systemd/system. Listing 18.2 gives an example of the vsftpd.service unit file.

Listing 18.2 Example of the Vsftpd Unit File

[Unit]
Description=Vsftpd ftp daemon
After=network.target

[Service]
Type=forking
ExecStart=/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf

[Install]
WantedBy=multi-user.target

From this unit file example, you can see that it is relatively easy to understand. Any systemd service unit file consists of three sections. (You’ll find different sections in other types of unit files.)

• [Unit], which describes the unit and defines dependencies. This section also contains the important After statement, and optionally the Before statement. These statements define dependencies between different units. The Before statement relates to another unit that is started after this unit. The after unit refers to a unit that needs to be started before this unit can be started.
• [Service], in which there is a description on how to start and stop the service and request status installation. Normally, you can expect an ExecStart line, which indicates how to start the unit, or an ExecStop line, which indicates how to stop the unit.
• [Install], in which the wants are taken care of. You’ll read more about this in the next section, “Understanding Target Units.”

Listing 18.3 shows another example of a unit file. This time it is the tmp.mount unit.

Listing 18.3 Example of a Mount Unit File

[Unit]
Description=Temporary Directory
Documentation=man:hier(7)
Documentation=http://www.freedesktop.org/wiki/Software/systemd/
  APIFileSystems
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target

[Mount]
What=tmpfs
Where=/tmp
Type=tmpfs
Options=mode=1777,strictatime

# Make 'systemctl enable tmp.mount' work:
[Install]
WantedBy=local-fs.target

The tmp.mount unit file in Listing 18.3 shows some interesting additional information. In the Unit section, you can see the Conflicts statement. This is used to list units that cannot be used together with this unit. Use this for mutually exclusive units. Next, there is the Mount section, which is specific for this unit type and defines where exactly the mount has to be performed. You’ll recognize the arguments that are typically used in any mount command. Last, there is the WantedBy section, which defines where the unit has to be started.

Another type of unit that is interesting to look at is the socket. A socket creates a method for applications to communicate with one another. Some services create their own sockets while starting, whereas other services need a socket unit file to create sockets for them. It is also the other way around: Every socket needs a corresponding service file. The socket file example in Listing 18.4 shows how this happens for virtlockd, a systemd socket that tracks activity for virtual machines.

Listing 18.4 Socket Unit File Example

[root@server202 system]# cat virtlockd.socket
[Unit]
Description=Virtual machine lock manager socket

[Socket]
ListenStream=/var/run/libvirt/virtlockd-sock

[Install]
WantedBy=multi-user.target

When working with systemd unit files, you risk getting overwhelmed with options. Every unit file can be configured with different options. To figure out which options are available for a specific unit, use the systemctl show command. For instance, the systemctl show sshd command shows all systemd options that can be configured in the sshd.service unit, including their current default values. Listing 18.5 shows the output of this command.

Listing 18.5 Showing Available Options with systemctl show

Id=sshd.service
Names=sshd.service
Requires=basic.target
Wants=sshd-keygen.service system.slice
WantedBy=multi-user.target
ConsistsOf=sshd-keygen.service
Conflicts=shutdown.target
ConflictedBy=sshd.socket
Before=shutdown.target multi-user.target
After=network.target sshd-keygen.service systemd-journald.socket
 basic.target system.slice
Description=OpenSSH server daemon
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/usr/lib/systemd/system/sshd.service
UnitFileState=enabled
InactiveExitTimestamp=Sat 2015-05-02 11:06:02 EDT
InactiveExitTimestampMonotonic=2596332166
ActiveEnterTimestamp=Sat 2015-05-02 11:06:02 EDT
ActiveEnterTimestampMonotonic=2596332166
ActiveExitTimestamp=Sat 2015-05-02 11:05:22 EDT
ActiveExitTimestampMonotonic=2559916100
InactiveEnterTimestamp=Sat 2015-05-02 11:06:02 EDT
InactiveEnterTimestampMonotonic=2596331238
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureIsolate=no
IgnoreOnIsolate=no

Understanding Target Units

The unit files are used to build the functionality that is needed on your server. To make it possible to load them in the right order and at the right moment, a specific type of unit is used: the target unit. A simple definition of a target unit is “a group of units.” Some targets are used as the equivalents to the old run levels, which on earlier versions of RHEL were used to define the state a server should be started in. A run level was a collection of services that were needed for a server to be started in multi-user mode or in graphical mode. Targets go beyond that. A good starting point to understanding targets is to see them as a group of units.

Targets by themselves can have dependencies to other targets, which are defined in the target unit. Let’s take a look at Listing 18.6, where you can see the definition of the multi-user.target, which defines the normal operational state of an RHEL server.

Listing 18.6 The Multi-user.target File

[root@server202 system]# cat multi-user.target
...

[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes

[Install]
Alias=default.target

You can see that by itself the target unit does not contain much. It just defines what it requires and which services and targets it cannot coexist with. It also defines load ordering, by using the After statement in the Unit section. And you can see that in the Install section it is defined as the default.target, so this is what your server starts by default. The target file does not contain any information about the units that should be included; that is in the individual unit files and the wants (explained in the upcoming section “Understanding Wants”).

Even if a systemd target looks a bit like the old run levels, it is more than that. A target is a group of units, and there are multiple different targets. Some targets, such as the multi-user.target and the graphical.target, define a specific state that the system needs to enter. Other targets just bundle a group of units together, such as the nfs.target and the printer.target. These targets are included from other targets, like the multi-user or graphical targets.

Understanding Wants

To understand the concept of a want, let’s start looking at the verb want in the English language, as in “I want a cookie.” Wants in systemd define which units systemd wants when starting a specific target. Wants are created when systemd units are enabled, and this happens by creating a symbolic link in the /etc/systemd/system directory. In this directory, you’ll find a subdirectory for every target, containing wants as symbolic links to specific services that are to be started.

Managing Units Through Systemd

As an administrator, you need to manage systemd units. It starts by starting and stopping units. You use the systemctl command to do that. In Exercise 18.1, you walk start, stop, and manage a unit. After you have configured a unit so that it can be started without problems, you need to make sure that it restarts automatically upon reboot. You do this by enabling or disabling the unit.

Listing 18.7 Requesting Current Unit Status with systemctl status

[root@server202 system]# systemctl status vsftpd
vsftpd.service - Vsftpd ftp daemon
   Loaded: loaded (/usr/lib/systemd/system/vsftpd.service; disabled)
   Active: active (running) since Sun 2014-09-28 08:42:59 EDT; 2s ago
  Process: 34468 ExecStart=/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf(code=exited, status=0/SUCCESS)
 Main PID: 34469 (vsftpd)
   CGroup: /system.slice/vsftpd.service
           ∟34469 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf

Sep 28 08:42:59 server202.example.com systemd[1]: Starting Vsftpd ftpdaemon...
Sep 28 08:42:59 server202.example.com systemd[1]: Started Vsftpd ftpdaemon.
Hint: Some lines were ellipsized, use -l to show in full.

When requesting the current status of a systemd unit as in Listing 18.6, you can see different kinds of information about it. Table 18.2 shows the different kinds of information that you can get about unit files when using the systemctl status command:

Table 18.2 Systemd Status Overview

As an administrator, you’ll also often need to get a current overview of the current status of systemd unit files. Different commands can help you to get this insight, some of which are shown in Table 18.3.

Table 18.3 Systemctl Unit Overview Commands

Managing Dependencies

Systemctl units in many cases have dependencies. Some units will be started as a dependency of other units, and an event where one specific unit is requested may trigger the start of another unit. An example is the cups.service service. This service can be started by itself, but it can also be started by activity on the cups.path and cups.socket units, which may trigger the service to be started again. As an administrator, you can request a list of unit dependencies. Type systemctl list-dependencies followed by a unit name to find out which dependencies it has, and add the --reverse option to find out which units are dependent of this unit. Listing 18.8 shows an example of this command.

Listing 18.8 Showing Unit Dependencies

[root@server1 ~]# systemctl list-dependencies vsftpd
    vsftpd.service
system.slice
∟basic.target
  alsa-restore.service
  alsa-state.service
  firewalld.service
  microcode.service
  rhel-autorelabel-mark.service
  rhel-autorelabel.service
  rhel-configure.service
  rhel-dmesg.service
  rhel-loadmodules.service
  paths.target
  slices.target
  |  -.slice
  |  ∟system.slice
  sockets.target
  |  avahi-daemon.socket
  |  cups.socket
  |  dbus.socket
  |  dm-event.socket
  |  iscsid.socket
  |  iscsiuio.socket
  |  lvm2-lvmetad.socket
  |  rpcbind.socket
  |  systemd-initctl.socket
  |  systemd-journald.socket
  |  systemd-shutdownd.socket
  |  systemd-udevd-control.socket
  |  ∟systemd-udevd-kernel.socket
  sysinit.target
  |  dev-hugepages.mount
  |  dev-mqueue.mount
  |  dmraid-activation.service
  |  iscsi.service

Apart from dependencies, some units have conflicts with other units. Examples of these include the following:

• Mount and umount units that cannot be loaded together
• The network and NetworkManager service
• The iptables and the firewalld service
• The cronyd and ntpd service

If units have conflicts with other units, this is described in the unit file. As an administrator, you can also make sure that conflicting units will never be loaded at the same time on the same system. To do this, you can use the systemctl mask command, which basically makes a unit no longer a candidate for being started. Apply the following procedure to find out how it works:

1. Open a root shell and type systemctl status firewalld. Next type systemctl status iptables. If one of the services is active, do not load it again in the next step.
2. Type systemctl start firewalld and systemctl start iptables to load both services. You will see that iptables refuses to start; this is because the firewalld service is already activated.
3. Type cat /usr/lib/systemd/system/firewalld.service. Notice the conflicts setting. Type cat /usr/lib/systemd/system/iptables.service. Notice that this unit does not have a conflicts line.
4. Unload both services by using systemctl stop firewalld followed by systemctl stop iptables. Notice that it is not really necessary to stop the iptables service because it has failed to load, but we really need to make sure that it is not loaded at all before continuing.

5. Type systemctl mask iptables and look at what is happening: A symbolic link to /dev/null is created for /etc/systemd/system/iptables.service (as you can see in the output of the following command example). Because the unit files in /etc/systemd have precedence over the files in /usr/lib/systemd, this makes it impossible to start the iptables service by accident:

   [root@server202 system]# systemctl mask iptables
   ln -s '/dev/null' '/etc/systemd/system/iptables.service'

6. Type systemctl start iptables. You’ll see an error message indicating that this service is masked and for that reason cannot be started.
7. Type systemctl enable iptables. Notice that no error message is shown and it looks as if it is working all right. Restart your server using systemctl reboot (or just reboot).

8. After restart, type systemctl status -l iptables. You’ll see that it is inactive and that the loaded status is indicated as masked:

   [root@server202 ~]# systemctl status -l iptables
   iptables.service
      Loaded: masked (/dev/null)
      Active: inactive (dead)

Managing Systemd Targets

As an administrator, you need to make sure that the required services are started when your server boots. To do this, use the systemctl enable and systemctl disable commands. You do not have to think about the specific target a service has to be started in. The services know for themselves in which targets they need to be started and a want is created automatically in that target. The following procedure walks you through the steps of enabling a service:

1. Type systemctl status vsftpd. If the service has not yet been enabled, the Loaded line will show that it currently is disabled:

   [root@server202 ~]# systemctl status vsftpd
   vsftpd.service - Vsftpd ftp daemon
      Loaded: loaded (/usr/lib/systemd/system/vsftpd.service;
     disabled)
      Active: inactive (dead)

2. Type ls /etc/systemd/system/multi-user.target.wants. You’ll see symbolic links that are taking care of starting the different services on your machine. You can also see that the vsftpd.service link does not exist.
3. Type systemctl enable vsftpd. The command shows you that it is creating a symbolic link for the file /usr/lib/systemd/system/vsftpd.service to the directory /etc/systemd/system/multi-user.target.wants. So basically, when you enable a systemd unit file, on the background a symbolic link is created.

Isolating Targets

As already discussed, on systemd machines there are a couple of targets. You also know that a target is a collection of units. Some of those targets have a special role because they can be isolated. By isolating a target, you start that target with all of its dependencies. Not all targets can be isolated, but only targets that have the isolate option enabled. We’ll explore the systemctl isolate command in a while. Before doing that, let’s take a look at the default targets on your computer.

To get a list of all targets currently loaded, type systemctl --type=target. You’ll see a list of all the targets currently active. If your server is running a graphical environment, this will include all the dependencies required to install the graphical.target also. However, this list does not show all the targets, but only the active targets. Type systemctl --type=target --all for an overview of all targets that exist on your computer. You’ll now see inactive targets also.

Of the targets on your system, a few have an important role because they can be started (isolated) to determine the state your server starts in. These are also the targets that can be set as the default target. These targets also roughly correspond to run levels as they were used on earlier versions of RHEL. These are the following targets:

• poweroff.target - runlevel 0
• rescue.target - runlevel 1
• multi-user.target - runlevel 3
• graphical.target - runlevel 5
• reboot.target - runlevel 6

If you look at the contents of each of these targets, you’ll also see that they contain the AllowIsolate=yes line. That means that you can switch the current state of your computer to either one of these targets using the systemctl isolate command. Exercise 18.2 shows you how to do this.

Setting the Default Target

Setting the default target is an easy procedure that can be accomplished from the command line. Type systemctl get-default to see the current default target and use systemctl set-default to set the desired default target.

To set the graphical target as the default target, you need to make sure that the required packages are installed. If this is not the case, you can use the yum group list command to show a list of all RPM package groups. The “server with gui” and “GNOME Desktop” package groups both apply. Use yum group install “server with gui” to install all GUI packages on a server where they have not been installed yet.

Working with GRUB 2

The GRUB 2 boot loader is one of the first things that needs to be working well to boot a Linux server. As an administrator, you will sometimes need to apply modifications to the GRUB 2 boot loader configuration. This section explains how to do so. The RHEL 7 boot procedure is discussed in more detail in Chapter 19, where troubleshooting topics are covered as well.

Understanding GRUB 2

The GRUB 2 boot loader makes sure that you can boot Linux. GRUB 2 is installed in the boot sector of your server’s hard drive and is configured to load a Linux kernel and the initramfs:

• The kernel is the heart of the operating system, allowing users to interact with the hardware that is installed in the server.
• The initramfs contains drivers that are needed to start your server. It contains a mini file system that is mounted during boot. In it are kernel modules that are needed during the rest of the boot process (for example, the LVM modules and SCSI modules for accessing disks that are not supported by default).

Normally, GRUB 2 works just fine and does not need much maintenance. In some cases, though, you might have to change its configuration. To apply changes to the GRUB 2 configuration, the starting point is the /etc/default/grub file. In this file, you’ll find options that tell GRUB what to do and how to do it. Listing 18.9 shows the contents of this file after an installation with default settings of CentOS 7.

Listing 18.9 Contents of the /etc/default/grub File

[root@server202 grub.d]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/swap vconsole.font=latarcyrheb-
  sun16 rd.lvm.lv=centos/root crashkernel=auto  vconsole.keymap=us
  rhgb quiet"
GRUB_DISABLE_RECOVERY="true"

As you can see, the /etc/default/grub file does not contain much information. The most important part that it configures is the GRUB-CMDLINE_LINUX option. This line contains boot arguments for the kernel on your server.

Apart from the configuration in /etc/default/grub, there are a few configuration files in /etc/grub.d. In these files, you’ll find rather complicated shell code that tells GRUB what to load and how to load it. You typically do not have to modify it. You also will not need to modify anything if you want to make it possible to select from different kernels while booting. GRUB 2 picks up new kernels automatically and adds them to the boot menu automatically, so nothing has to be added manually.

Based on the configuration files mentioned previously, the main configuration file /boot/grub2/grub.cfg is created. Even if this looks like a configuration file that can be manually modified, you should never do that, because it will get overwritten at some point. This will happen, for instance, after updating the kernel. The RPM from which the kernel is updated will run a post-installation script that regenerates the kernel. In the next section, you learn how to make changes to the GRUB 2 configuration.

If you enter the GRUB 2 boot prompt to add kernel startup parameters, the contents of the /boot/grub2/grub.cfg file display. From here, you add one-time-only startup options. Listing 18.10 shows the relevant part of the grub.cfg file that takes care of loading the Linux kernel. In this listing, you see the part of the configuration file that takes care of loading the default kernel. Notice the line that starts with linux16; this line specifies all kernel boot parameters.

Listing 18.10 Partial Contents of the /boot/grub2/grub.cfg Configuration File

menuentry 'CentOS Linux (3.10.0-229.1.2.el7.x86_64) 7 (Core)' --class
centos --class gnu-linux --class gnu --class os --unrestricted
$menuentry_id_option 'gnulinux-3.10.0-123.el7.x86_64-advanced-50faa2a1-
01d3-430b-8114-4a98daf5bdb9' {
      load_video
      set gfxpayload=keep
      insmod gzio
      insmod part_msdos
      insmod xfs
      set root='hd0,msdos1'
      if [ x$feature_platform_search_hint = xy ]; then
        search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1
--hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'
057ba3d8-bfe7-4676-bb99-79e9980a1966
     else
       search --no-floppy --fs-uuid --set=root 057ba3d8-bfe7-4676-bb99-79e9980a1966
     fi
     linux16 /vmlinuz-3.10.0-229.1.2.el7.x86_64 root=/dev/mapper/
centos-root ro rd.lvm.lv=centos/swap vconsole.font=latarcyrheb-sun16
rd.lvm.lv=centos/root crashkernel=auto  vconsole.keymap=us rhgb quiet
LANG=en_US.UTF-8
     initrd16 /initramfs-3.10.0-229.1.2.el7.x86_64.img
}

One of the most important differences between GRUB 2 and its previous version is the availability of GRUB 2 modules. In GRUB 2, a large number of modules are available. By default, you can find them in /boot/grub2/i386-pc. The modules determine what you can and what you cannot do from the GRUB 2 boot loader. If some hardware or file system is not supported in what you want to do, check here to make sure that a supporting GRUB 2 module is available. In Listing 18.10, you can see examples of the code used to load specific GRUB 2 modules.

Modifying Default GRUB 2 Boot Options

To apply modifications to the GRUB 2 boot loader, the file /etc/default/grub is your entry point; do not change the contents of the /boot/grub2/grub.cfg configuration file directly. The most important line in this file is GRUB_CMDLINE_LINUX, which defines how the Linux kernel should be started. In this line, you can apply permanent fixes to the GRUB 2 configuration. Some likely candidates for removal are the options rhgb and quiet. These options tell the kernel to hide all output while booting. That is nice to hide confusing messages for end users, but if you are a server administrator, you probably just want to remove these options.

Another interesting parameter is GRUB_TIMEOUT. This defines the amount of time your server waits for you to access the GRUB 2 boot menu before it continues booting automatically. If your server runs on physical hardware that takes a long time to get through the BIOS checks, it may be interesting to increase this time a bit.

While working with GRUB 2, you need to know a bit about kernel boot arguments. There are many of them, and most of them you’ll never use, but it is good to know where you can find them. Type man 7 bootparam for a man page that contains an excellent description of all boot parameters that you may use while starting the kernel.

In Exercise 18.3, you learn how to apply modifications to GRUB 2.

Summary

In this chapter you learned how systemd and GRUB 2 are used to bring your server into the exact state you desire at the end of the boot procedure. You also learned how systemd is organized, and also how units can be configured for automatic start with the use of targets. You also read how to apply changes to the default GRUB 2 boot loader. In the next chapter, you learn how to troubleshoot the boot procedure and fix some common problems.