Pipes & redirection

The Unix philosophy is "write programs that do one thing well, and work together". The glue that makes them work together is redirection (connect a command to files) and pipes (connect commands to each other). Once these click, you'll be building one-line tools that would take a page of code elsewhere.

The three standard streams#

Every process starts with three open "files", known by number (file descriptors):

Both stdout and stderr appear on your screen, but they are separate streams. That's why you can save results to a file and still see errors.

Redirecting output: > and >>#

echo "first line" > notes.txt      # create or OVERWRITE
echo "second line" >> notes.txt    # APPEND
cat notes.txt

first line
second line

⚠️ > truncates the file before the command even runs. sort data.txt > data.txt leaves you with an empty file! Write to a new file, or use sort -o data.txt data.txt (or sponge from moreutils).

Turn on set -o noclobber to make > refuse to overwrite existing files (>| forces it).

Redirecting errors: 2>#

ls notes.txt missing.txt > out.txt 2> err.txt
echo "--- out.txt"; cat out.txt
echo "--- err.txt"; cat err.txt

--- out.txt
notes.txt
--- err.txt
ls: cannot access 'missing.txt': No such file or directory

Common patterns:

ls notes.txt missing.txt > all.txt 2>&1    # both streams to one file (portable)
ls notes.txt missing.txt &> all2.txt       # same thing, bash shorthand
ls missing.txt 2> /dev/null                # throw errors away
cat all.txt

ls: cannot access 'missing.txt': No such file or directory
notes.txt

/dev/null is the "black hole" device: anything written to it disappears, and reading from it gives an immediate end-of-file.

Order matters. cmd > file 2>&1 sends both to the file. cmd 2>&1 > file first points stderr at the terminal (where stdout currently goes) and only then moves stdout to the file, so errors still appear on screen.

Redirecting input: <, here-docs and here-strings#

wc -l < notes.txt                 # feed a file to stdin
tr 'a-z' 'A-Z' <<< "shout this"   # here-string: a single string as stdin
cat <<EOF2
Hello $USER,
today is $(date +%A).
EOF2

2
SHOUT THIS
Hello ada,
today is Thursday.

A here-document (<<WORD ... WORD) feeds several lines to a command, which is great for writing config files from scripts. Variables and $(...) are expanded inside it; quote the marker (<<'EOF') to keep the text literal. Use <<-EOF to strip leading tabs so you can indent it inside scripts.

Pipes: connecting commands#

A pipe | connects the stdout of one command to the stdin of the next:

printf "banana\napple\ncherry\napple\nbanana\napple\n" > fruit.txt
cat fruit.txt | sort | uniq -c | sort -rn

      3 apple
      2 banana
      1 cherry

Read it left to right: list, sort (so duplicates are adjacent), count duplicates, sort by count descending. All the commands in a pipeline run at the same time, streaming data, so pipelines work on files far bigger than memory.

A few classic pipelines:

ls /usr/bin | wc -l                          # how many commands are installed?
ps aux --sort=-%mem | head -5                # top memory users
history | awk '{print $2}' | sort | uniq -c | sort -rn | head   # your most-used commands
du -sh /var/log/* 2>/dev/null | sort -h | tail -5               # biggest items in /var/log

Only stdout goes through a pipe. To pipe errors too, use 2>&1 | or the bash shorthand |&:

ls notes.txt missing.txt 2>&1 | grep -c "No such"

1

tee: save and see#

tee writes its input to a file and passes it on:

echo "build ok" | tee build.log
echo "tests ok" | tee -a build.log > /dev/null
cat build.log

build ok
build ok
tests ok

tee also solves the classic sudo redirection problem. In sudo echo x > /etc/file, the redirection is done by your shell, which isn't root. Instead:

echo "net.ipv4.ip_forward=1" | sudo tee /etc/sysctl.d/99-forward.conf
echo "extra line" | sudo tee -a /etc/some.conf > /dev/null

Command substitution: $( )#

$(command) runs a command and inserts its output into another command line:

echo "Today is $(date +%Y-%m-%d) and there are $(wc -l < fruit.txt) fruits"
backup="notes-$(date +%Y%m%d).txt"
cp notes.txt "$backup" && ls notes-*

Today is 2026-10-01 and there are 6 fruits
notes-20261001.txt

You'll see old scripts use backticks `date`. They do the same job, but $( ) nests cleanly and is easier to read.

xargs: turn input into arguments#

Some commands don't read stdin; they want arguments (rm, mkdir, cp...). xargs reads items from stdin and passes them as arguments:

printf "a.log\nb.log\nc.log\n" | xargs touch
ls *.log
echo "a.log b.log" | xargs -n 1 echo "deleting:"
find . -name "?.log" -print0 | xargs -0 rm -v

a.log  b.log  build.log  c.log
deleting: a.log
deleting: b.log
removed './b.log'
removed './a.log'
removed './c.log'

• -n 1 runs the command once per item.
• -I {} lets you place the item anywhere: xargs -I {} cp {} {}.bak.
• -P 4 runs up to 4 commands in parallel.
• -print0 / -0 separate items with NUL bytes, so filenames with spaces or newlines are safe. Always use this pair with find.

Exit codes and chaining#

Every command finishes with an exit status: 0 means success, anything else is failure. $? holds the last one:

grep -q apple fruit.txt; echo "exit: $?"
grep -q mango fruit.txt; echo "exit: $?"
mkdir -p build && echo "created" || echo "failed"
false; echo "after false: $?"

exit: 0
exit: 1
created
after false: 1

A pipeline's exit status is the last command's, so false | true succeeds. In scripts, set -o pipefail makes a pipeline fail if any part fails (covered in the scripting module).

Process substitution (bash)#

<(command) makes a command's output look like a file, which is handy for tools that want filenames:

diff <(printf "a\nb\nc\n") <(printf "a\nB\nc\n")

2c2
< b
---
> B

Common mistakes#

• cmd > file on the file you're reading: the file is truncated before reading starts.
• 2>&1 in the wrong place: cmd 2>&1 > file still prints errors to the screen.
• sudo cmd > /root-owned/file: use | sudo tee file.
• Parsing ls output in pipelines: filenames with spaces break it. Use globs or find -print0 | xargs -0.
• Useless cat: cat file | grep x works, but grep x file is simpler. That's harmless, but good to know.
• Forgetting that pipes carry only stdout: add 2>&1 (or |&) if you need errors too.

What's next#

Pipes are most powerful with good filters. Next comes text processing: cut, sort, uniq, tr, wc, paste and friends.