Standard IO streams in Linux — Tech Tip Bits

Standard IO refers to standardized input, output, and error streams in Linux that can be read and written by all standard Linux utilities. It’s sometimes abbreviated as “STDIO”, while standard input is called STDIN, standard output as STDOUT, and the standard error stream is called as STDERR.

STDIO operations in the Linux shell

Most Linux commands support standard io instead of using files and they also default to the standard input / output when there is no filename given to work on in the command line parameters.

Simply running stdio aware commands will result in waiting for input from the shell (i.e. user input) and output will be written to the standard output, i.e. the shell will display the output of the command instantly.

For example, the command “sort” has the following support for file i/o:

sort -o outputfile inputfile

Both the output file and the input file are optional.

STDIO examples using Bash

Running the command “sort” without any parameters waits for the user input, then it processes all the input and outputs its result to the stdout (the current shell display). The keyboard command CTRL+D terminates line input.

$ sort tech tip bits <CTRL-D> bits tech tip

Here, typing “sort<enter>tech<enter>tip<enter>bits<ctrl-d>” ran the sort command, gave it the input of three lines containing tech/tip/bits, then terminated input by pressing ctrl-d. Sort did its thing and shows the result to the terminal — the three lines now sorted.

Not all commands wait for the whole input to be completed before output starts, for example, the grep command by default looks for a match and displays matching lines only. It means that while receiving data from the standard input, it either silently “eats” lines or displays them again:

$ grep i tech tip tip <-- this is the output of grep bits bits <-- this is the outout of grep

The command above looks for the letter “i” and only displays lines that contain that letter. As lines are being entered, the line that matches is instantly displayed. This is different from the output of sort because sort needs to read the whole input before sorting, while grep can instantly decide if a line needs to be sent to stdout (because it matches) or not.

IO redirection using files

Both the standard input and the output can be redirected using the “<” and “>” shell redirection operators. These operators need to be appended to the command and they should be followed by the filename they refer to.

For example, to send a file to the standard input of a command, it should be written as

command < file

In the same way, to redirect the output of a command to a file, it should be written as

command > file

These can also be combined, so to feed a file into a command then save its output elsewhere, we could write it as

command < inputfile > outputfile

Let’s look at an example, create a file that contains the lines “tech, tip, bits”:

$ cat>input.txt tech tip bits <ctrl-d>

This creates the file input.txt and it will contain these three lines. Now to sort it, we can use the following command:

$ sort < input.txt bits tech tip

To save the output into a file, we can simply redirect the output of the sort command instead of displaying it:

$ sort < input.txt > output.txt $ cat output.txt bits tech tip

This, by the way, is equivalent to typing the one below, because sort supports setting both the input and the output file using command line parameters:

sort -o output.txt input.txt

Chaining commands using the pipe operator

Until now, we have only redirected stdio streams to/from files, but we can also directly connect commands I/O to each other, by using the “|” (pipe) operator. This is the fundamental building block of Linux command chains and it’s one of the most important tools to implement complete workflows using multiple commands.

For example, if we’d like to find characters in a file, then sort results, we would do this without redirection:

$ grep i input.txt > output.txt $ sort -i output.txt bits tip

Grep doesn’t support output files so we can only save its output by redirecting the output. However, we can combine the two commands by using the pipe operator:

$ grep i input.txt | sort bits tip

It’s also possible to combine pipes with multiple file redirections, here grep’s input is sent from a file and sort’s output is sent to another file and the two commands are connected using a pipe:

$ grep i < input.txt | sort > output.txt $ cat output.txt bits tip

The standard error stream

In addition to the input and the output stream, there is a third one, called standard error or STDERR. This is an output stream where all error messages are written to. By default, it points to the same terminal output as the standard output. Redirecting the standard output leaves stderr pointing to the terminal which is quite handy because it means that error messages are displayed even when the output is redirected to a file.

$ sort notfound.txt > output.txt sort: No such file or directory

After running this command, output.txt will be empty (0 bytes) and the error message is still displayed to the terminal.

It’s possible to redirect the standard error stream to a file, by using the “2>” operator:

$ sort notfound.txt > output.txt 2> error.txt $ cat error.txt sort: No such file or directory

One useful trick is that it’s possible to send the stderr into stdout and save them to a common file by doing this:

$ sort notfound.txt > output.txt 2>&1 $ cat output.txt sort: No such file or directory $ sort input.txt > output.txt 2>&1 $ cat output.txt bits tech tip

This way both the output and the error is saved to a common file.

Conclusion

Using shell redirection operators it’s possible to create complicated command chains using multiple files and connecting different commands to each other. These streams are often drop-in replacements for files unless seeking is required because a program won’t be able to rewind or jump to the end of a stream like it would be able to read arbitrary parts of a file using direct i/o.

Originally published at https://techtipbits.com on May 29, 2021.