Standard IO streams in Linux — Tech Tip Bits

Standard IO refers to standardized input, output, and error streams in Linux that can be read and written by all standard Linux utilities. It’s sometimes abbreviated as “STDIO”, while standard input is called STDIN, standard output as STDOUT, and the standard error stream is called as STDERR.

STDIO operations in the Linux shell

Most Linux commands support standard io instead of using files and they also default to the standard input / output when there is no filename given to work on in the command line parameters.

Simply running stdio aware commands will result in waiting for input from the shell (i.e. user input) and output will be written to the standard output, i.e. the shell will display the output of the command instantly.

For example, the command “sort” has the following support for file i/o:

Both the output file and the input file are optional.

STDIO examples using Bash

Running the command “sort” without any parameters waits for the user input, then it processes all the input and outputs its result to the stdout (the current shell display). The keyboard command CTRL+D terminates line input.

Here, typing “sort<enter>tech<enter>tip<enter>bits<ctrl-d>” ran the sort command, gave it the input of three lines containing tech/tip/bits, then terminated input by pressing ctrl-d. Sort did its thing and shows the result to the terminal — the three lines now sorted.

Not all commands wait for the whole input to be completed before output starts, for example, the grep command by default looks for a match and displays matching lines only. It means that while receiving data from the standard input, it either silently “eats” lines or displays them again:

The command above looks for the letter “i” and only displays lines that contain that letter. As lines are being entered, the line that matches is instantly displayed. This is different from the output of sort because sort needs to read the whole input before sorting, while grep can instantly decide if a line needs to be sent to stdout (because it matches) or not.

IO redirection using files

Both the standard input and the output can be redirected using the “<” and “>” shell redirection operators. These operators need to be appended to the command and they should be followed by the filename they refer to.

For example, to send a file to the standard input of a command, it should be written as

In the same way, to redirect the output of a command to a file, it should be written as

These can also be combined, so to feed a file into a command then save its output elsewhere, we could write it as

Let’s look at an example, create a file that contains the lines “tech, tip, bits”:

This creates the file input.txt and it will contain these three lines. Now to sort it, we can use the following command:

To save the output into a file, we can simply redirect the output of the sort command instead of displaying it:

This, by the way, is equivalent to typing the one below, because sort supports setting both the input and the output file using command line parameters:

Chaining commands using the pipe operator

Until now, we have only redirected stdio streams to/from files, but we can also directly connect commands I/O to each other, by using the “|” (pipe) operator. This is the fundamental building block of Linux command chains and it’s one of the most important tools to implement complete workflows using multiple commands.

For example, if we’d like to find characters in a file, then sort results, we would do this without redirection:

Grep doesn’t support output files so we can only save its output by redirecting the output. However, we can combine the two commands by using the pipe operator:

It’s also possible to combine pipes with multiple file redirections, here grep’s input is sent from a file and sort’s output is sent to another file and the two commands are connected using a pipe:

The standard error stream

In addition to the input and the output stream, there is a third one, called standard error or STDERR. This is an output stream where all error messages are written to. By default, it points to the same terminal output as the standard output. Redirecting the standard output leaves stderr pointing to the terminal which is quite handy because it means that error messages are displayed even when the output is redirected to a file.

After running this command, output.txt will be empty (0 bytes) and the error message is still displayed to the terminal.

It’s possible to redirect the standard error stream to a file, by using the “2>” operator:

One useful trick is that it’s possible to send the stderr into stdout and save them to a common file by doing this:

This way both the output and the error is saved to a common file.

Conclusion

Using shell redirection operators it’s possible to create complicated command chains using multiple files and connecting different commands to each other. These streams are often drop-in replacements for files unless seeking is required because a program won’t be able to rewind or jump to the end of a stream like it would be able to read arbitrary parts of a file using direct i/o.

Originally published at https://techtipbits.com on May 29, 2021.

Information Technology Simplified