Tee and Xargs: Splitting and Feeding
tee saves what flows through a pipe into a file and passes it on, so you can keep a report and read it at once, and xargs turns the lines it reads into arguments for commands such as mkdir and rm, which don't read a pipe.
Journeyman · Unix · Linux · Updated
After this lesson you can
- Save a pipeline's output in a file and still see it with tee, and add to a file with tee -a
- Run a command on the lines a pipeline prints with xargs, checking first with xargs echo
Key takeaways
- tee saves what comes through a pipe into a file and also passes it on, to the screen or the next command. tee -a adds to the end of the file instead of replacing it.
- Commands such as mkdir and rm take names as arguments and ignore a pipe. xargs reads the lines and runs the command with them as its arguments.
- Put echo in front of the command, as in xargs echo rm, to see the command xargs would run without running it.
- xargs splits at spaces as well as at line ends, so a file name with a space becomes two names.
Bea's accountant wants the city ranking in a file, and Bea wants to read it now, too. With > she gets the file and a blank screen, and then has to cat it. And her receipts folder holds a receipt for every birthday cake, a few of them for orders that were canceled: she wants exactly those receipts gone. grep -l finds them, but rm doesn't read a pipe. Two more plumbing fittings solve both: tee splits the flow in two, and xargs feeds lines to a command that wants names. IT techs and developers use them to keep a record of a job while watching it, and to act on exactly the files a search found.
The examples use the bakery folder from Pipes: | (its setup lines make it), and cities.txt from Count, Sort, Remove Duplicates: wc, sort and uniq. Go into the folder with cd ~/terminal-practice/bakery.
tee: save it and see it
A T-shaped joint in a water pipe splits the flow two ways. tee (read "tee", like the letter T) does the same with lines: it writes them into a file and passes them on:
cut -d, -f5 orders.csv | tail -n +2 | sort | uniq -c | sort -rn | tee city-report.txt
212 Portland
96 Salem
89 Bend
73 Eugene
27 Medford
23 Corvallis
The ranking showed on the screen, and the same six lines are in the file:
wc -l city-report.txt
6 city-report.txt
Like >, tee replaces the file each time. tee -a (for append) adds to its end, like >>, so this adds the bestsellers under the cities:
cut -d, -f6 orders.csv | tail -n +2 | sort | uniq -c | sort -rn | tee -a city-report.txt
wc -l city-report.txt
175 sourdough loaf
100 croissant
80 blueberry muffin
78 baguette
68 cinnamon roll
19 birthday cake
12 city-report.txt
Because tee passes its lines on, it can also sit in the middle of a pipeline and save a stage on the way past. This saves the mailing list from Columns and Characters: cut and tr and counts it, in one line:
cut -d, -f4 orders.csv | tail -n +2 | tr 'A-Z' 'a-z' | sort -u | tee mailing-list.txt | wc -l
20
xargs: lines into arguments
Bea wants a folder for each city, to keep its delivery notes in. sort -u cities.txt lists the six cities, but mkdir takes folder names as arguments, the words after its name, and never reads a pipe:
sort -u cities.txt | mkdir -p
error: the following required arguments were not provided:
<dirs>...
Usage: mkdir [OPTION]... DIRECTORY...
For more information, try '--help'.
However it's worded on your system, the message means the same thing: mkdir got no folder names. xargs (read "x args") reads the lines that come in, puts them after a command as its arguments and runs it. Before you let it run anything, put echo in front of the command. xargs then prints the command it would run, and runs nothing:
sort -u cities.txt | xargs echo mkdir -p
mkdir -p Bend Corvallis Eugene Medford Portland Salem
That's the command you'd have typed by hand. Take echo away to run it:
sort -u cities.txt | xargs mkdir -p
ls -d */
Bend/ Corvallis/ Eugene/ Medford/ Portland/ Salem/
Delete exactly what a search found
These lines make the receipts folder, with a receipt for each of seven cake orders, then mark two of them canceled. The for line repeats one echo for each order number; For Every File: for Loops, in Module 5, teaches loops. They're safe to run again.
mkdir -p receipts
for n in 1025 1054 1103 1185 1463 1499 1514; do echo "Order $n: birthday cake, PAID" > receipts/cake-$n.txt; done
echo "Order 1103: birthday cake, CANCELED" > receipts/cake-1103.txt
echo "Order 1463: birthday cake, CANCELED" > receipts/cake-1463.txt
ls receipts
cake-1025.txt cake-1103.txt cake-1463.txt cake-1514.txt
cake-1054.txt cake-1185.txt cake-1499.txt
rm deletes for good, so follow the habit from Deleting Safely: rm and rmdir: look first. grep -l (from Finding Text: grep) lists the files that contain a word:
grep -l CANCELED receipts/*
receipts/cake-1103.txt
receipts/cake-1463.txt
Then see the command xargs would build:
grep -l CANCELED receipts/* | xargs echo rm
rm receipts/cake-1103.txt receipts/cake-1463.txt
Exactly the two canceled receipts. Now run it, and check what's left:
grep -l CANCELED receipts/* | xargs rm
ls receipts
cake-1025.txt cake-1054.txt cake-1185.txt cake-1499.txt cake-1514.txt
One warning: xargs splits what it reads at spaces as well as at line ends, so a file named cake order.txt would reach rm as two names, cake and order.txt. When names might have spaces, let find end each name with an invisible marker, -print0, and tell xargs to split only there with -0: find receipts -name "*.txt" -print0 | xargs -0 ls.
Check yourself
2 questions to try on the lesson page.
So what
With tee you never have to choose between saving a result and seeing it, and with xargs any list a pipeline makes can drive any command: make, move or delete exactly the files you found.