Linux Piping and Redirection Explained

Every program which is run on the command line has three data streams and Linux piping and redirection allows them to be redirected.

The three data streams are:

• STDIN (0) - Standard input
• STDOUT (1) - Standard output
• STDERR (2) - Standard error

Let's look at each of these and how we can manipulate the streams to write files or send one program's output to another's input.

STDOUT

STDOUT is used to redirect output to a file. Redirecting output to a file will overwrite existing data if the file already exists. This can be handy when sending the listings output to a file for printing or emailing. Redirecting STDOUT to a file is as easy as using the greater than symbol.

ls -l > output.txt

A double greater than (>>) is also used to redirect output to the file. However, this method will append the data to an existing file.

ls -l >> output.txt

It can be easy to overwrite files using output redirection, as there are no warnings. However, you can use the noclobber shell variable to prevent overwriting of files from output redirection.

You can look at the shell options using shell -o; we can see that by default, noclobber is off. We can turn this on by entering set -o noclobber and turn it off again with set +o noclobber. To set this automatically when logging in, you can add this to your .bashrc file.

When noclobber is enabled, you are prevented from overwriting files. If you wish to overwrite, you must force an overwrite by including a vertical bar in the redirect.

ls -l >| output.txt

STDERR

STDERR is similar to STDOUT but is used to handle error messages instead.

A simple way of demonstrating the difference between STDERR and STDOUT is to try to redirect the output of a command that generates an error.

ls /etcw
ls: cannot access /etcw: no such file or directory

We can then try and redirect this to a file:

ls /etcw > error.txt
ls: cannot access /etcw: no such file or directory

Note that you can still see the error displayed on the screen. If you look at the contents of the error.txt, you will note that it is empty. The STDOUT was redirected to the file, while the error message was sent to the console using a different pipe. By default, the greater than symbol output STDOUT only and since error messages are sent to STDERR, they were not redirected. We can redirect the error output to a log file by specifying the pipe when redirecting the output. This is done by entering the pipe number before the operator.

ls /etcw 2> error.txt

Piping Inputs

We've seen how to send output to a file, but we can also read the contents of a file and send it to a program's input. The output from one command can be piped directly into another command for further processing. This is done with the less-than-symbol symbol. We can use the diskfree command to show the available disk space on the system.

df -hlT

We could pipe this to an output file using the output pipe.

df -hlT > diskfree.txt

But in this example, I want to send this data as an email to myself. As a system administrator, I want weekly usage reports and to be able to take proactive action in the event of low disk space.

We can read the contents of the text file and pipe it to the mail command, which will read the STDIN and send the text file as an attachment.

mail -s "Disk Free" tim < diskfree.txt

This will then read in the file and send the contents as the body text of the mail message.

Here Documents

A here document is a special-purpose code block. It uses a form of I/O redirection to feed a command list to an interactive program or a command. They are very useful for creating small text files.

Using cat piped to a new file; we can then use a here document to read input directly into the new file.

cat > newfile <<END

You can now type text content into the file until you type in END. You don't have to use end; you could use something like EOF or anything else that wouldn't be in the input.

cat > newfile <<END
This is a sample text file
The cat sat on the mat
END

cat newfile

This is a sample text file.
The cat sat on the mat

Command Pipelines

Many of the little commands we have seen are designed to be plumbed together using pipes. That is, the output from one command is directly fed into another command.

We've already seen one example when we piped the output of the find command to grep.

sudo apt --installed list | grep apache

We can also get a count of line numbers in a directory listing you can use

ls | wc -l
26

To examine the password file to get a count of logins. The password file lists the user logins along with the shell used.

head -n1 /etc/passwd

root:x:0:0:root:/root:/bin/bash

We may want to see the number of unique shell types in use.

We can see that the passwd is a : separated file.

We can use the cut command to split the line at :; the shell command is the 7th field.

cut -f7 -d: /etc/passwd
/bin/bash
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/bin/sync
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/bin/false

Here is a list of all the default shells. We can sort this output.

cut -f7 -d: /etc/passwd | sort

And then remove duplicate entries.

cut -f7 -d: /etc/passwd | sort | unique

Now, we can get a count of unique entries as well.

cut -f7 -d: /etc/passwd | sort | unique | wc -l

Named Pipes

Named pipes are used when allowing different processes to talk to each other. They don't hold any data; they merely marshal the data between two processes. Named pipes have a special file type of p, which can be seen on the metadata block of ls -l

ls -l 
prw-r--r-- 1 timmy timmy         0 Aug 16 20:28 mynewpipe

Creating a pipe is done with the mkfifo command.

mkfifo mynewpipe

In this example, we'll see what happens when writing to this pipe in one SSH session and reading it in another.

In terminal session 1, you can send output to the pipe the same as we have seen before

ls -l > mynewpipe

In session 2, you read the output in much the same way as before

wc -l < mynewpipe

Please note: The execution thread, when sending it to the pipe, will be locked until the data has been read out from the pipe. This means that if there is no service or command pulling data from the pipe, commands or scripts sending data into the pipe will be blocked until the pipe is cleared.

Teeing Off

When we pipe to a file or other command, you cannot see the output on the screen. The output was redirected from stdout to the file or command.

However, we can use the tee command to split the screen output and the output file or command.

ls > listing.txt
[no outpt]

ls | tee listing.txt
drwxr-xr-x 2 timmy timmy      4096 Jun 24 18:01 Desktop
drwxr-xr-x 2 timmy timmy      4096 Jul  8 21:08 Documents
drwxr-xr-x 2 timmy timmy     12288 Aug 16 19:49 Downloads
drwxr-xr-x 2 timmy timmy      4096 Sep  3  2016 Music

This will write the listing in the text file and show the listing on the screen.