This chapter is from the book
This chapter is from the book
This chapter is from the book
Processing the Command Line
Whether you are working interactively or running a shell script, bash needs to read a command line before it can start processing it—bash always reads at least one line before processing a command. Some bash builtins, such as if and case, as well as functions and quoted strings, span multiple lines. When bash recognizes a command that covers more than one line, it reads the entire command before processing it. In interactive sessions bash prompts you with the secondary prompt (PS2, > by default; page 288) as you type each line of a multiline command until it recognizes the end of the command:
$ echo 'hi
> end'
hi
end
$ function hello () {
> echo hello there
> }
$
After reading a command line, bash applies history expansion and alias substitution to the line.
History Expansion
"Reexecuting and Editing Commands" on page 296 discusses the commands you can give to modify and reexecute command lines from the history list. History expansion is the process that bash uses to turn a history command into an executable command line. For example, when you give the command !!, history expansion changes that command line so it is the same as the previous one. History expansion is turned on by default for interactive shells; set +o histexpand turns it off. History expansion does not apply to noninteractive shells (shell scripts).
Alias Substitution
Aliases (page 311) substitute a string for the first word of a simple command. By default aliases are turned on for interactive shells and off for noninteractive shells. Give the command shopt –u expand_aliases to turn aliases off.
Parsing and Scanning the Command Line
After processing history commands and aliases, bash does not execute the command immediately. One of the first things the shell does is to parse (isolate strings of characters in) the command line into tokens or words. The shell then scans each token for special characters and patterns that instruct the shell to take certain actions. These actions can involve substituting one word or words for another. When the shell parses the following command line, it breaks it into three tokens (cp, ~/letter, and .):
$ cp ~/letter .
After separating tokens and before executing the command, the shell scans the tokens and performs command line expansion.
Command Line Expansion
In both interactive and noninteractive use, the shell transforms the command line using command line expansion before passing the command line to the program being called. You can use a shell without knowing much about command line expansion, but you can use what a shell has to offer to a better advantage with an understanding of this topic. This section covers Bourne Again Shell command line expansion; TC Shell command line expansion is covered starting on page 341.
The Bourne Again Shell scans each token for the various types of expansion and substitution in the following order. Most of these processes expand a word into a single word. Only brace expansion, word splitting, and pathname expansion can change the number of words in a command (except for the expansion of the variable "$@"—page 566).
- Brace expansion (page 323)
- Tilde expansion (page 324)
- Parameter and variable expansion (page 325)
- Arithmetic expansion (page 325)
- Command substitution (page 327)
- Word splitting (page 328)
- Pathname expansion (page 329)
- Process substitution (page 330)
Quote removal
After bash finishes with the preceding list, it removes from the command line single quotation marks, double quotation marks, and backslashes that are not a result of an expansion. This process is called quote removal.
Order of expansion
The order in which bash carries out these steps affects the interpretation of commands. For example, if you set a variable to a value that looks like the instruction for output redirection and then enter a command that uses the variable's value to perform redirection, you might expect bash to redirect the output.
$ SENDIT="> /tmp/saveit" $ echo xxx $SENDIT xxx > /tmp/saveit $ cat /tmp/saveit cat: /tmp/saveit: No such file or directory
In fact, the shell does not redirect the output—it recognizes input and output redirection before it evaluates variables. When it executes the command line, the shell checks for redirection and, finding none, evaluates the SENDIT variable. After replacing the variable with > /tmp/saveit, bash passes the arguments to echo, which dutifully copies its arguments to standard output. No /tmp/saveit file is created.
The following sections provide more detailed descriptions of the steps involved in command processing. Keep in mind that double and single quotation marks cause the shell to behave differently when performing expansions. Double quotation marks permit parameter and variable expansion but suppress other types of expansion. Single quotation marks suppress all types of expansion.
Brace expansion
Brace expansion, which originated in the C Shell, provides a convenient way to specify filenames when pathname expansion does not apply. Although brace expansion is almost always used to specify filenames, the mechanism can be used to generate arbitrary strings; the shell does not attempt to match the brace notation with the names of existing files.
Brace expansion is turned on in interactive and noninteractive shells by default; you can turn it off with set +o braceexpand. The shell also uses braces to isolate variable names (page 281).
The following example illustrates how brace expansion works. The ls command does not display any output because there are no files in the working directory. The echo builtin displays the strings that the shell generates with brace expansion. In this case the strings do not match filenames (there are no files in the working directory.)
$ ls
$ echo chap_{one,two,three}.txt
chap_one.txt chap_two.txt chap_three.txt
The shell expands the comma-separated strings inside the braces in the echo command into a SPACE-separated list of strings. Each string from the list is prepended with the string chap_, called the preamble, and appended with the string .txt, called the postscript. Both the preamble and the postscript are optional. The left-to-right order of the strings within the braces is preserved in the expansion. For the shell to treat the left and right braces specially and for brace expansion to occur, at least one comma and no unquoted whitespace characters must be inside the braces. You can nest brace expansions.
Brace expansion is useful when there is a long preamble or postscript. The following example copies the four files main.c, f1.c, f2.c, and tmp.c located in the /usr/local/src/C directory to the working directory:
$ cp /usr/local/src/C/{main,f1,f2,tmp}.c .
You can also use brace expansion to create directories with related names:
$ ls -F
file1 file2 file3
$ mkdir vrs{A,B,C,D,E}
$ ls -F
file1 file2 file3 vrsA/ vrsB/ vrsC/ vrsD/ vrsE/
The –F option causes ls to display a slash (/) after a directory and an asterisk (*) after an executable file.
If you tried to use an ambiguous file reference instead of braces to specify the directories, the result would be different (and not what you wanted):
$ rmdir vrs* $ mkdir vrs[A-E] $ ls -F file1 file2 file3 vrs[A-E]/
An ambiguous file reference matches the names of existing files. Because it found no filenames matching vrs[A–E], bash passed the ambiguous file reference to mkdir, which created a directory with that name. Page 136 has a discussion of brackets in ambiguous file references.
Tilde expansion
Chapter 4 (page 78) showed a shorthand notation to specify your home directory or the home directory of another user. This section provides a more detailed explanation of tilde expansion.
The tilde (~) is a special character when it appears at the start of a token on a command line. When it sees a tilde in this position, bash looks at the following string of characters—up to the first slash (/) or to the end of the word if there is no slash—as a possible login name. If this possible login name is null (that is, if the tilde appears as a word by itself or if it is immediately followed by a slash), the shell substitutes the value of the HOME variable for the tilde. The following example demonstrates this expansion, where the last command copies the file named letter from Alex's home directory to the working directory:
$ echo $HOME /Users/alex $ echo ~ /Users/alex $ echo ~/letter /Users/alex/letter $ cp ~/letter .
If the string of characters following the tilde forms a valid login name, the shell substitutes the path of the home directory associated with that login name for the tilde and name. If it is not null and not a valid login name, the shell does not make any substitution:
$ echo ~jenny /Users/jenny $ echo ~root /var/root $ echo ~xx ~xx
Tildes are also used in directory stack manipulation (page 275). In addition, ~+ is a synonym for PWD (the name of the working directory), and ~– is a synonym for OLDPWD (the name of the previous working directory).
Parameter and Variable Expansion
On a command line a dollar sign ($) that is not followed by an open parenthesis introduces parameter or variable expansion. Parameters include command line, or positional, parameters (page 564) and special parameters (page 562). Variables include user-created variables (page 279) and keyword variables (page 284). The bash man and info pages do not make this distinction, however.
Parameters and variables are not expanded if they are enclosed within single quotation marks or if the leading dollar sign is escaped (preceded with a backslash). If they are enclosed within double quotation marks, the shell expands parameters and variables.
Arithmetic Expansion
The shell performs arithmetic expansion by evaluating an arithmetic expression and replacing it with the result. See page 355 for information on arithmetic expansion under tcsh. Under bash the syntax for arithmetic expansion is
$ ((
expression
))
The shell evaluates expression and replaces $(( expression )) with the result of the evaluation. This syntax is similar to the syntax used for command substitution [$(...)] and performs a parallel function. You can use $(( expression )) as an argument to a command or in place of any numeric value on a command line.
The rules for forming expression are the same as those found in the C programming language; all standard C arithmetic operators are available (see Table 13-8 on page 588). Arithmetic in bash is done using integers. Unless you use variables of type integer (page 284) or actual integers, however, the shell must convert string-valued variables to integers for the purpose of the arithmetic evaluation.
You do not need to precede variable names within expression with a dollar sign ($). In the following example, an arithmetic expression determines how many years are left until age 60:
$ cat age_check #!/bin/bash echo -n "How old are you? " read age echo "Wow, in $((60-age)) years, you'll be 60!" $ age_check How old are you? 55 Wow, in 5 years, you'll be 60!
You do not need to enclose the expression within quotation marks because bash does not perform filename expansion on it. This feature makes it easier for you to use an asterisk (*) for multiplication, as the following example shows:
$ echo There are $((60*60*24*365)) seconds in a non-leap year. There are 31536000 seconds in a non-leap year.
The next example uses wc, cut, arithmetic expansion, and command substitution to estimate the number of pages required to print the contents of the file letter.txt. The output of the wc utility (page 888) used with the –l option is the number of lines in the file, in columns 1 through 4, followed by a SPACE and the name of the file (the first command following). The cut utility (page 699) with the –c1-4 option extracts the first four columns.
$ wc -l letter.txt 351 letter.txt $ wc -l letter.txt | cut -c1-4 351
The dollar sign and single parenthesis instruct the shell to perform command substitution; the dollar sign and double parentheses indicate arithmetic expansion:
$ echo $(( $(wc -l letter.txt | cut -c1-4)/66 + 1)) 6
The preceding example sends standard output from wc to standard input of cut via a pipe. Because of command substitution, the output of both commands replaces the commands between the $( and the matching ) on the command line. Arithmetic expansion then divides this number by 66, the number of lines on a page. A 1 is added at the end because the integer division results in any remainder being discarded.
Another way to get the same result without using cut is to redirect the input to wc instead of having wc get its input from a file you name on the command line. When you redirect its input, wc does not display the name of the file:
$ wc -l < letter.txt
351
It is common practice to assign the result of arithmetic expansion to a variable:
$ numpages=$(($(wc -l < letter.txt)/66 + 1))
let builtin
The let builtin (not available in tcsh) evaluates arithmetic expressions just as the $(( )) syntax does. The following command is equivalent to the preceding one:
$ let "numpages=$(wc -l < letter.txt)/66 + 1"
The double quotation marks keep the SPACEs (both those you can see and those that result from the command substitution) from separating the expression into separate arguments to let. The value of the last expression determines the exit status of let. If the value of the last expression is 0, the exit status of let is 1; otherwise, the exit status is 0.
You can give multiple arguments to let on a single command line:
$ let a=5+3 b=7+2 $ echo $a $b 8 9
When you refer to variables when doing arithmetic expansion with let or $(()), the shell does not require you to begin the variable name with a dollar sign ($). Nevertheless, it is a good practice to do so, as in most places you must include this symbol.
Command Substitution
Command substitution replaces a command with the output of that command. The preferred syntax for command substitution under bash follows:
$(
command
)
Under bash you can also use the following syntax, which is the only syntax allowed under tcsh:
'command'
The shell executes command within a subshell and replaces command , along with the surrounding punctuation, with standard output of command .
In the following example, the shell executes pwd and substitutes the output of the command for the command and surrounding punctuation. Then the shell passes the output of the command, which is now an argument, to echo, which displays it.
$ echo $(pwd) /Users/alex
The next script assigns the output of the pwd builtin to the variable where and displays a message containing the value of this variable:
$ cat where where=$(pwd) echo "You are using the $where directory." $ where You are using the /Users/jenny directory.
Although it illustrates how to assign the output of a command to a variable, this example is not realistic. You can more directly display the output of pwd without using a variable:
$ cat where2 echo "You are using the $(pwd) directory." $ where2 You are using the /Users/jenny directory.
The following command uses find to locate files with the name README in the directory tree with its root at the working directory. This list of files is standard output of find and becomes the list of arguments to ls.
$ ls -l $(find . -name README -print)
The next command line shows the older 'command' syntax:
$ ls -l 'find . -name README -print'
One advantage of the newer syntax is that it avoids the rather arcane rules for token handling, quotation mark handling, and escaped back ticks within the old syntax. Another advantage of the new syntax is that it can be nested, unlike the old syntax. For example, you can produce a long listing of all README files whose size exceeds the size of ./README with the following command:
$ ls -l $(find . -name README -size +$(echo $(cat ./README | wc -c)c ) -print )
Try giving this command after giving a set –x command (page 536) to see how bash expands it. If there is no README file, you just get the output of ls –l.
For additional scripts that use command substitution, see pages 532, 549, and 579.
Word Splitting
The results of parameter and variable expansion, command substitution, and arithmetic expansion are candidates for word splitting. Using each character of IFS (page 288) as a possible delimiter, bash splits these candidates into words or tokens. If IFS is unset, bash uses its default value (SPACE-TAB-NEWLINE). If IFS is null, bash does not split words.
Pathname Expansion
Pathname expansion (page 133), also called filename generation or globbing, is the process of interpreting ambiguous file references and substituting the appropriate list of filenames. Unless noglob (page 320) is set, the shell performs this function when it encounters an ambiguous file reference—a token containing any of the unquoted characters *,?, [, or ]. If bash cannot locate any files that match the specified pattern, the token with the ambiguous file reference is left alone. The shell does not delete the token or replace it with a null string but rather passes it to the program as is (except see nullglob on page 320). The TC Shell generates an error message.
In the first echo command in the following example, the shell expands the ambiguous file reference tmp* and passes three tokens (tmp1, tmp2, and tmp3) to echo. The echo builtin displays the three filenames it was passed by the shell. After rm removes the three tmp* files, the shell finds no filenames that match tmp* when it tries to expand it. Thus it passes the unexpanded string to the echo builtin, which displays the string it was passed.
$ ls tmp1 tmp2 tmp3 $ echo tmp* tmp1 tmp2 tmp3 $ rm tmp* $ echo tmp* tmp*
By default the same command causes the TC Shell to display an error message:
tcsh $ echo tmp* echo: No match
A period that either starts a pathname or follows a slash (/) in a pathname must be matched explicitly unless you have set dotglob (page 319). The option nocaseglob (page 320) causes ambiguous file references to match filenames without regard to case.
Quotation marks
Putting double quotation marks around an argument causes the shell to suppress pathname and all other expansion except parameter and variable expansion. Putting single quotation marks around an argument suppresses all types of expansion. The second echo command in the following example shows the variable $alex between double quotation marks, which allow variable expansion. As a result the shell expands the variable to its value: sonar. This expansion does not occur in the third echo command, which uses single quotation marks. Because neither single nor double quotation marks allow pathname expansion, the last two commands display the unexpanded argument tmp*.
$ echo tmp* $alex tmp1 tmp2 tmp3 sonar $ echo "tmp* $alex" tmp* sonar $ echo 'tmp* $alex' tmp* $alex
The shell distinguishes between the value of a variable and a reference to the variable and does not expand ambiguous file references if they occur in the value of a variable. As a consequence you can assign to a variable a value that includes special characters, such as an asterisk (*).
Levels of expansion
In the next example, the working directory has three files whose names begin with letter. When you assign the value letter* to the variable var, the shell does not expand the ambiguous file reference because it occurs in the value of a variable (in the assignment statement for the variable). No quotation marks surround the string letter*; context alone prevents the expansion. After the assignment the set builtin (with the help of grep) shows the value of var to be letter*.
The three echo commands demonstrate three levels of expansion. When $var is quoted with single quotation marks, the shell performs no expansion and passes the character string $var to echo, which displays it. When you use double quotation marks, the shell performs variable expansion only and substitutes the value of the var variable for its name, preceded by a dollar sign. No pathname expansion is performed on this command because double quotation marks suppress it. In the final command, the shell, without the limitations of quotation marks, performs variable substitution and then pathname expansion before passing the arguments to echo.
$ ls letter* letter1 letter2 letter3 $ var=letter* $ set | grep var var='letter* $ echo '$var' $var $ echo "$var" letter* $ echo $var letter1 letter2 letter3
Process Substitution
A special feature of the Bourne Again Shell is the ability to replace filename arguments with processes. An argument with the syntax <(command) causes command to be executed and the output written to a named pipe (FIFO). The shell replaces that argument with the name of the pipe. If that argument is then used as the name of an input file during processing, the output of command is read. Similarly an argument with the syntax >(command) is replaced by the name of a pipe that command reads as standard input.
The following example uses sort (page 837) with the –m (merge, which works correctly only if the input files are already sorted) option to combine two word lists into a single list. Each word list is generated by a pipe that extracts words matching a pattern from a file and sorts the words in that list.
$ sort -m -f <(grep "[^A-Z]..$" memo1 | sort) <(grep ".*aba.*" memo2 |sort)