Linux Piping and Redirection Explained

Every program which is run on the command line has three data streams and Linux piping and redirection allows them to be redirected.

Every program which is run on the command line has three data streams and piping and redirection can be used to connect these streams between programs and files to direct data in interesting and useful ways.

The three data streams are:

• STDIN (0) - Standard input
• STDOUT (1) - Standard output
• STDERR (2) - Standard error

Let's have a look at each of these and how we can manipulate the streams to write files or send the output of one program to the input of another.

STDOUT

STDOUT is used to redirect output to a file. Redirecting output to a file will overwrite existing data if the file already exists. This can be handy to send the output of listings to a file for printing or emailing. Redirecting STDOUT to a file is as easy as using the greater than symbol.

ls -l > output.txt

There is also a double greater than (>>) which is also used to redirect output to the file, however, this method will append the data to an existing file.

ls -l >> output.txt

It can be easy to overwrite files using output redirection as there are no warnings, however, you can use the noclobber shell variable to prevent overwriting of files from output redirection.

You can look at the shell options using shell -o, we can see that by default noclobber is off. We can turn this on by entering set -o noclobber, and turn it off again with set +o noclobber. To set this automatically at login you can add this to your .bashrc file.

When noclobber is enabled, you are prevented from overwriting files. If you so wish to overwrite, you have to force an overwrite by including a vertical bar in the redirect.

ls -l >| output.txt

STDERR

STDERR is similar to STDOUT but it is used to handle error messages instead.

A simple way of demonstrating the difference between STDERR and STDOUT is to try and redirect the output of a command which generates an error.

ls /etcw
ls: cannot access /etcw: no such file or directory

We can then try and redirect this to a file:

ls /etcw > error.txt
ls: cannot access /etcw: no such file or directory

Note, you can still see the error displayed on the screen. If you look at the contents of the error.txt, you will note that it is empty. This is because the STDOUT was redirected to the file, while the error message was sent to the console using a different pipe. By default, the greater than symbol output STDOUT only and since error messages are sent to STDERR, they were not redirected. We can redirect the error output to a log file by specifying the pipe when redirecting the output. This is done by entering the pipe number before the operator.

ls /etcw 2> error.txt

Piping Inputs

We've seen how to send output to a file, but we can also read the contents of a file and send it to a program's input. The output from one command can be piped directly into another command for further processing. This is done with the less than symbol symbol. We can use the disk free command to show the available disk space on the system

df -hlT

We could pipe this to an output file using the output pipe

df -hlT > diskfree.txt

But in this example, I want to be able to send this data as an email to myself. As a system administrator I want weekly usage reports and to be able to take proactive action in the event of low disk space.

We can read the contents of the text file and pipe it to the mail command which will read the STDIN and send the text file as an attachment.

mail -s "Disk Free" tim < diskfree.txt

This will then read in the file and send the contents as the body text of the mail message.

Here Documents

A here document is a special-purpose code block. It uses a form of I/O redirection to feed a command list to an interactive program or a command. They are very useful for creating small text files.

Using cat piped to a new file we can then use a here document to read input directly into the new file

cat > newfile <<END

You can now type text content into the file until you type in END. You don't have to use end, you could use something like EOF or anything else that wouldn't be in the input itself.

cat > newfile <<END
This is a sample text file
The cat sat on the mat
END

cat newfile

This is a sample text file
The cat sat on the mat

Command Pipelines

A lot of the little commands we have seen are designed to be plumbed together using pipes. That is the output from one command is directly fed into another command.

We've already seen one example when we piped the output of the find command to grep.

sudo apt --installed list | grep apache

We can also get a count of line numbers in a directory listing you can use

ls | wc -l
26

To examine the passwords file to get a count of logins. The passwords file lists the user logins along with the shell used.

head -n1 /etc/passwd

root:x:0:0:root:/root:/bin/bash

Maybe we want to see the number of unique shell types that are in use.

We can see that the passwd is a : separated file

We can use the cut command to split the line at : and that the shell command is the 7th field

cut -f7 -d: /etc/passwd
/bin/bash
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/bin/sync
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin
/bin/false

Here is a list of all the default shells. We can sort this output

cut -f7 -d: /etc/passwd | sort

And then remove duplicate entries

cut -f7 -d: /etc/passwd | sort | unique

Now we can get a count of unique entries as well

cut -f7 -d: /etc/passwd | sort | unique | wc -l

Named Pipes

Named pipes are used when allowing different processes to talk to each other. They don't hold any data, they merely marshal the data between two processes. Named pipes have a special file type of p which can be seen on the metadata block of ls -l

ls -l 
prw-r--r-- 1 timmy timmy         0 Aug 16 20:28 mynewpipe

Creating a pipe is done with the mkfifo command.

mkfifo mynewpipe

In this example, we'll see what happens when writing to this pipe in one ssh session and reading it in another.

In terminal session 1, you can send output to the pipe the same as we have seen before

ls -l > mynewpipe

In session 2 you read the output in much the same way as before

wc -l < mynewpipe

Please note: The execution thread on the sending to the pipe will be locked until the data has been read out from the pipe. This means that if there is no service or command pulling data from the pipe, commands or scripts sending data into the pipe will be blocked until the pipe is cleared.

Teeing Off

As you may have noticed, when we pipe to a file or other command you cannot see the output on the screen. This is because the output was redirected away from stdout to the file or command.

We can, however, use the tee command to split the output of the screen and the output file or command.

ls > listing.txt
[no outpt]

ls | tee listing.txt
drwxr-xr-x 2 timmy timmy      4096 Jun 24 18:01 Desktop
drwxr-xr-x 2 timmy timmy      4096 Jul  8 21:08 Documents
drwxr-xr-x 2 timmy timmy     12288 Aug 16 19:49 Downloads
drwxr-xr-x 2 timmy timmy      4096 Sep  3  2016 Music

This will write the listing to the text file as well as show the listing on the screen.