Want to support this site? The Command Line for Web Developers is available to purchase on Amazon in print and Leanpub in most digital formats. Sales from the book help maintain and update this site and future editions. Reader support is available on Leanpub’s discussion page.

3. More Advanced Bash

Here’s a question for you: how is the command line like jQuery?

If you were a developer like me before we had smart phones and when every browser behaved wildly different from each other, then you know writing JavaScript was not easy. However, jQuery gave us the ability to write scripts without having to know too much about JavaScript. jQuery is JavaScript, but it has an easy-to-understand API and structure, which makes it easy to build elaborate scripts with just a few of its functions.

You don’t have to know all of jQuery to use it. And if you come across new jQuery functions, it isn’t difficult to understand them because the library has a clear and simple syntax.

Like jQuery, many of the commands used in Bash have a shared structure. They take arguments similarly, they have the same inputs and outputs, and they (hopefully) well-document the weirder parts.

Unlike jQuery, there is too much information on Bash to cover here. To go through all of the commands in Bash and all of their flags in addition to covering the complete shell script syntax would warrant a book all its own. There are hundreds of commands in Bash,¹ and we will be adding even more to the mix as we get farther into this book. Instead of focusing on each command, we will address the most common ones and how they operate in the context of the Bash syntax.

The goal is that once you understand how these popular commands work and how Bash commands are structured, you will be able to recognize the syntax structure for other commands. These command examples should let you break down code we haven’t covered. And just like learning a new function in jQuery, the new code will just be a matter of vocabulary.

Finding data with grep

Almost all of Bash commands’ names are abbreviated descriptions of their functions. cp is short for copy. Also ls is short for list, mv is short for move, and so on. grep is not as apparent in what it stands for: globally search a regular expression and print. A regular expression, or regex, is a pattern of characters to describe what we look for in searches that don’t fit in typical string searches. For instance, when we write a web page that asks for an email address, typically our script will use a regular expression to test if that email address is valid. Regular expressions are used not just for JavaScript but for other programming languages as well. This command will return search results from a file or group of files.

In Figure 3-1, I used a regular expression to return any line of text that had a “G” at the beginning of a line (^ means beginning of a line).

Regular expressions are incredibly useful, and it is worth understanding how they work not only for Bash but for other programs and programming languages.

grep is used in a similar way to cp and mv in its command syntax. cp and mv two took two arguments: the first was the file to be acted upon, and the second was where it was to go. grep also takes two arguments but, in this case, the first argument is the search pattern, and the second is the file to search. Like cp and mv, grep also has useful flags operators as well. The -n flag returns line numbers with the results.

Can you tell what we searched for in Figure 3-3? Looking at the results you can probably tell that '\bW\w*\b' returns all words that begin with “W.” If we break down the string of characters used for the search, it reads like this: find a “W” character with any word boundary (\b) before it: it can have any number of word characters (\w*) after it and ends with another word boundary (\b).

A word boundary can be a space, the end of a line of text, or the beginning of a line of text. Simply put, it is any character that qualifies as a separator between words. A word character is any character that can be used to form words. When we use the regex pattern in this example, we are searching for all US presidents that have a name (first, middle, or last) that starts with “W.”

The results show seven presidents. Because we used the -n flag in our command, we can see the line number the presidents’ names occur in the file.

When we replace the last part of the expression with $ we change the statement from looking for all words that begin with “J” to looking for only those that also are the last word in a line. By doing this, we only search for last names that start with “J.”

Searching for presidents is fine if you need to win a trivia question, but we work with code across multiple files and directories.

Let’s say that you are editing one of your websites, and you need to know which files are using a class name .btn in your CSS files. Remember that grep uses regular expressions. Special characters like . need to be escaped or else grep will interpret it differently, and you will get the wrong results. In regex, . means any single character, so it isn’t just another string to find; it has a pattern meaning. Putting a \ (backslash) in front of .btn escapes it from that meaning, and the whole thing becomes a string character. Let’s step through what this command says in Figure 3-5. Do a grep search for any occurrence for the string “.btn” in this current directory, and do it recursively and return the line numbers as well.

Now wait a minute! We used \ in pattern searches in previous examples like \b and \w , so how does using \. make it a character search? To make it more confusing, we use . at the very end! The simple answer is that regex is weird! The more complicated answer is that some characters are reserved for regex, like . and *, while alphabetic characters are not. We are using the backslash to change escape them (or change their meaning). Unless you intend to become an expert at regular expressions, my advice is to keep a list of patterns that you come across that you can reuse.

One thing we expect from searching is to be able to see our search terms highlighted in the context of the results. Adding color to grep will highlight the term in each result.

What the terminal printed shows us which files have our match and the line numbers within those files. We can narrow down our search parameters even more. Instead of returning all files with the found match, let’s say we are only interested in JavaScript files. In this case, we’d use something like what is shown in Figure 3-7.

The --include flag is really helpful when working in a project with a hefty number of files, many of which are precompiled. If you were searching for a specific CSS style, it might not be useful to know where your match exists in CSS when you intend to make an edit in Sass.

Using pipes to chain commands

Picking up from our last scenario, let’s build on a command that just searched within Sass files.

$ grep -rn --color=always --include \*.scss '\.btn' .

Depending on your project, the number of results that come back will vary. .btn might be a highly referenced class name. If you work in Bootstrap, a common class name that gets used extensively will make the grep results scroll off the screen. This time we are going to pipe our search results to another command called less.

Now, do not get this confused with the compiler language for CSS called LESS. The command less is completely different. It lets you read a file one “page” at a time. When less is running, you can use the space bar, cursor keys, or your mouse wheel/track pad to scroll through the results at your own pace. Using less to read one page at a time is much more digestible than reading potentially hundreds of lines of text scrolling off the page.

Figure 3-8 used two commands together, separated by a pipe character, |.

A pipe lets you take one command’s output and feed it to another command. A good metaphor is to think about how plumbing works. A pipe at one end has the grep command “pouring” its output of search results into the pipe and the less command is on the other end of the pipe taking that as its input.

Pipes can be chained together to use multiple commands. Let’s say you are working on a site that is using too many different colors, and you want to know which ones are being used so you can consolidate them into a simpler palette. The example in Figure 3-9 uses | to identify any qualifying hex color pattern in variables.scss, then sort the results, and then show only the unique values from that entire list.

The command uses -E for extended regular expressions. Yes, there are two forms of regular expressions that grep can use (the first version of regex did not include the ability to use the pattern in this example). In this case the expression commands to look for both short and long forms of CSS hex values (CSS accepts both #FFF and #FFFFFF). Then the returned results are piped to the sort command. Finally, that new result is piped to uniq which only works with sorted items because it looks at the adjacent value.

One more pipe example: let’s assume that variables.scss is the only file defining colors. In this case we don’t have to worry about CSS color words or RGA/RGBA values. In addition to seeing a list of all of the colors, we want to know how many times those values occur throughout the file. We want to clean up this file, so let’s get those repeat offenders.

We are working with the same command as before; we are just adding more pipes to it (Figure 3-10). For uniq, we are adding a count flag to know how many duplicates it found. This puts a number value in front of the values returned. This output is then piped to sort again, but now we are sorting numerically (-n) and in reverse order (-r). All of this is piped to less so we can view the results one page at a time.

Sometimes it isn’t always clear what the intent of a chain of piped commands is supposed to accomplish or how the final result came to be. Try running these commands one at a time, each time adding another piece of the chain. That way you can better understand how it all fits together by seeing what is output from one pipe to the next.

Other ways to chain commands

Pipes are one type of redirect that make commands more versatile. You can chain commands that will run after the other has completed by using ; and && (double ampersand).

Both of these accomplish the same thing in this case. Both will change to your home directory and then list everything in that directory. So what’s the difference? Commands separated by a semicolon will always execute in order. Commands separated by the double ampersand will break on error.

In Figure 3-12, the directory sandbox already exists, so the first command fails. From the previous chapter, you should know that you can’t remove a directory with just rm, so that command fails too. And the last command would have confirmed sandbox was no longer there. All three of these commands still executed when using semicolons even though two of them had errors.

Using the double ampersand instead of a semicolon will let the commands break at the first error.

One way to remember the difference between ; and && is how we use this syntax in JavaScript. Semicolons are at the end of of a statement in JavaScript. Running a JavaScript file will step through each statement, and even though some things will break, a lot of times the file will continue to execute. Double ampersands are used to evaluate two expressions in JavaScript. So if you can remember that double ampersands evaluate, and semicolons just move along that should help you remember the difference.

Some Bash syntax is easy to remember (like cd stands for change directory). Others are more challenging when they are so closely related. Finding common meanings with other languages like JavaScript can help you remember them more easily.

The double pipe separator, ||, is how we run one command only if the previous command fails.

In Figure 3-13, if bucket.txt doesn’t exist, we create the file.

Redirection syntax

Redirection is a way to change what happens to the input and output of a command. We already used some redirection earlier in the chapter. So far, all of our command output ends up printing to the screen. Most of the time, this is the default output of a command. But we can change the output to files instead by using redirects (Figure 3-14).

If you run this command, you will see no output printed to the screen. Instead, you saved it to a file called list.txt. You can verify the file with cat:

$ cat list.txt

cat will print the file list.txt to the screen. What prints should look just like the output from the ls command. > is how we redirect the stdout (standard output) of a command. Before, when we were using ls, the stdout was what printed to the screen.

In addition to stdout, many commands have a stdin (standard input), which is marked with <. These are the same inputs we used in the pipe examples from before, but we didn’t need to call out < explicitly. Most of the time, this is the case.

Common redirection examples

Below are a few other types of redirections. Some are more common than others, and we will have examples of them throughout the book. This table isn’t a complete list, but it does illustrate the numerous types of redirections available.

A well-written command, whether it is in Bash, Ruby, or Node, will have both a stdout and a stderr (standard error). Sass, a CSS extension language, will output both success messages and error messages to stdout, but all of the debugging info is sent to stderr. This is good for when you need to see how a function is outputting values.

@function em ($px, $font-size: $font-size) {
  @return ($px / $font-size) * 1em;
  @degug ($n);
}

Let’s say you have a commonly used mixin, and you want to know how it is being used and what values it returns. Because Sass has a debug feature, you will see debug info print to the screen.

While this information is important for debugging, it can feel a bit “noisy” to have all that information printing each time you compile. This is especially true when you are only interested in knowing if your CSS files successfully compiled or if an error occurred.

In (Figure 3-15), we are using Sass to watch for file changes, but we are sending any stderr messages to a special Unix file called /dev/null. This file is special because it is always empty no matter what you send to it. It is a “null device” and can be considered a black hole for data. In this case, it is a great way to ignore debugging information so you can focus on your task of writing great CSS code.

/dev/null is also sometimes called a “bit bucket.” which goes back to a literal bucket that collected chads from punch cards used in computer programming.

Shortcuts to accessing desktop applications

In Finder, you can make an alias to another file, folder, or application by dragging an icon to a new location while holding down the command key and option key together. In Bash, we can do similar actions with different strategies.

Symbolic Links

A symbolic link, or soft link, is just like a shortcut in Finder. It’s a reference to another file or directory.

In the first command in Figure 3-16, we set the target and the reference and then print the list to see what we created. As you can see, the first column of info, which holds info about permissions, begins with l to denote the type of file it is (a link). And we can see where this link points at the end of the line.

Symbolic links can be useful when you have one source of files that need to be read by multiple projects. If you need to change directory to a location that is long to type out, for example, a symbolic link to that directory would be very useful. But if the original gets deleted or moved to another location, this link will no longer resolve.

Application helpers

Some desktop applications come with their own command line program. Sublime Text, a very popular text editor, has one such program. By using ln like in Figure 3-17, we can open files and directories in Sublime Text from the command line.

In Figure 3-17, we’ve created a symbolic link (subl) to point to /Applications/Sublime Text.app/Contents/SharedSupport/bin/subl. Instead of having to enter in the full path to that command, we now only have to type “subl.”

In order for Bash to start using subl as a command, we created a bin directory in our home directory and set our PATH variable to include bin in its value. We will go into more detail about why this is a good idea, but essentially PATH is the global variable that knows where to look for all of Bash’s commands.

Windows users should note that to use Sublime Text in the command line, instead of making a symbolic link we need to go about this differently. But it’s still possible when using the Windows command line (start > run > cmd) like in Figure 3-19.

Now you can use subl as a command and point to any file or directory, and it will open in Sublime Text. Other Desktop Applications do this too,² and some will offer to install this ability for you, such as in Brackets.

Aliases

Unlike aliases in Finder, Bash aliases are shortcuts for whole commands. All of the examples we’ve gone over so far can be aliased to tiny shortcuts. This is incredibly handy when you repeat a command over and over.

In example Figure 3-20, I almost always prefer to use ls with these flags. -a shows all files, -l prints in the long format, and -F adds which files are actually directories (folders). I set an alias to handle these tasks for me. Using ll by itself works just as it does when passing in your own arguments as you would with the original ls command.

In example Figure 3-21, we want to know the number of words in all the MarkDown documents in this directory. We could go through each file individually, run the wc command, and then add them up. However, in the example listed above, we find all of the files we are looking for at once simply by using find. Because the result prints each file on a new line, we use tr to separate them by a space instead. The string ‘ ’ means a new line, and tr finds all instances of that argument and replaces it with ‘ ’ (a space).

Now that we have all of the files listed in a single line, separated by spaces, cat will concatenate all of the files into output, but we need to use xargs to make sure that cat is reading the input from tr as proper arguments and not just a string of text. And then we get just the word count with wc and the -w (word count only) flag.

If the functionality of xargs doesn’t make sense, try running the command with it and just useing cat by itself, without xargs. What you will see is cat outputting just the names of the files passed to it and not the content of those files. xargs is useful in making sure the command’s input is interpreted properly.

You can have complex commands like this one saved to an easy-to-remember alias. Notice how the single quotes and double quotes have to obey the same syntax rules used in JavaScript.

$ alias rm='rm -i'

You can also alias a command to itself. Here we are setting rm to always run with the “interactive” flag so we don’t unintentionally delete something. A good guide to making rm less of a risk³ is to always be aware that your aliases and shortcuts only exist in your environment. Should you ever have to go to another computer, it will most definitely not have your aliases.

When you start using the command line on a regular basis, you are going to find your own set of commands that would be a good to have as an alias.

Searching command history

One other way to get to commands faster is to search for them. We learned that Bash keeps a history of executed commands, and we can browse them by using the cursor keys in the command prompt. It may be faster to search for a command using the key stroke ctrl-r. Use the ctrl (control) key and r key to bring up the search prompt.

As shown in Figure 3-22, reverse-i-search performs grep-like searches in real time as you begin typing. This command will display the first result it finds in the prompt. If what it finds is what you are looking for, simply hit the escape key, and you will return to your normal Bash prompt with that command ready to go. If that is not the command you are looking for, you can either refine your search by adding more characters, or you can cycle through the current results by hitting the control and r keys until you find the command you want.

By default, Bash keeps a history of 500 of your executed commands. Having a way like ctrl-r to search them is very useful.

PATH is where commands are located

Earlier we went over modifying the PATH variable to include ~/bin using export. We can examine the contents of that variable just like we did with HOME.

These are paths that Bash will look for when you type in a command. In other words, commands like cd and git exist as in one of these directories. By having ~/bin in this list, we can include our own commands. The advantage of this is that it doesn’t affect the operating system by adding miscellaneous commands to directories like /usr/bin that the OS controls.

We see in Figure 3-24 that cd is a command program that has a location that Bash refers to every time we use that command. Bash knows the command is in the directory because the PATH variable has /usr/bin as a value. And because we made a symbolic link to a directory that is also in the PATH variable, Bash knows where it is located; thus, when we call cd or subl in a command, they will execute.

Note that the ll alias does not return anything. Aliases are the exception to this. The way to check if an alias exists is simply to enter the command alias:

$ alias
alias ll='ls -alF'

You’re a wizard, Harry

Bash uses many commands and syntaxes. We’ve covered several of them already and learned how to combine them to expand their functionality to get the results we want. We will continue to add to the list of Bash commands throughout the book. There is an index in the back of the book for reference as well.

If you’re still not feeling comfortable using the command line, that’s totally fine. It takes practice, and there is a lot to learn. Nobody becomes an expert overnight at this any more than one does with web design or web development. Learn one thing at a time, and then add another thing to that. Before long, you will be “casting” your own commands.

1. http://ss64.com/bash/
2. Chapter 8: Text Editors
3. https://github.com/sindresorhus/guides/blob/master/how-not-to-rm-yourself.md