Shell by Example: Line Filters POSIX

Line filters read from stdin, transform data, and write to stdout. The while read pattern is the foundation of shell line processing. Here’s a basic while read loop that reads from stdin. The -r flag prevents backslash interpretation. Always use quotes around $line to preserve whitespace.
Edit 
#!/bin/sh
echo "Basic line reading:"
printf "line1\nline2\nline3\n" | while read -r line; do
    echo "  Got: $line"
done
Output:
Basic line reading:
  Got: line1
  Got: line2
  Got: line3

Process a file line by line:
Edit 
#!/bin/sh
cat >/tmp/data.txt <<'EOF'
apple 10 red
banana 20 yellow
cherry 15 red
date 25 brown
EOF

echo "Processing file:"
while read -r fruit count color; do
    echo "  $fruit ($color): $count items"
done </tmp/data.txt
Output:
Processing file:
  apple (red): 10 items
  banana (yellow): 20 items
  cherry (red): 15 items
  date (brown): 25 items

IFS controls field splitting:
Edit 
#!/bin/sh
echo "Custom delimiter (CSV):"
printf "alice,30,engineer\nbob,25,designer\n" | while IFS=, read -r name age job; do
    echo "  $name is $age, works as $job"
done
Output:
Custom delimiter (CSV):
  alice is 30, works as engineer
  bob is 25, works as designer

Read from command output:
Edit 
#!/bin/sh
echo "Reading from command:"
find /tmp -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null | head -5 | while read -r line; do
    echo "  $line"
done
Output:
Reading from command:

Filter pattern: transform each line
Edit 
#!/bin/sh
echo "Transform each line (uppercase):"
printf "hello\nworld\n" | while read -r line; do
    echo "$line" | tr '[:lower:]' '[:upper:]'
done
Output:
Transform each line (uppercase):
HELLO
WORLD

Numbering lines:
Edit 
#!/bin/sh
echo "Number lines:"
n=1
printf "first\nsecond\nthird\n" | while read -r line; do
    printf "%3d: %s\n" "$n" "$line"
    n=$((n + 1))
done
Output:
Number lines:
  1: first
  2: second
  3: third

Skip header line:
Edit 
#!/bin/sh
echo "Skip header:"
cat >/tmp/csv.txt <<'EOF'
name,age,city
Alice,30,NYC
Bob,25,LA
Carol,35,Chicago
EOF

head -1 /tmp/csv.txt
tail -n +2 /tmp/csv.txt | while IFS=, read -r name age city; do
    echo "  $name (age $age) from $city"
done
Output:
Skip header:
name,age,city
  Alice (age 30) from NYC
  Bob (age 25) from LA
  Carol (age 35) from Chicago

Filter out empty lines:
Edit 
#!/bin/sh
echo "Skip empty lines:"
printf "line1\n\nline2\n\nline3\n" | while read -r line; do
    [ -z "$line" ] && continue
    echo "  $line"
done
Output:
Skip empty lines:
  line1
  line2
  line3

Filter with grep before processing:
Edit 
#!/bin/sh
cat >/tmp/data.txt <<'EOF'
apple 10 red
banana 20 yellow
cherry 15 red
date 25 brown
EOF

grep "red" /tmp/data.txt | while read -r fruit count color; do
    echo "  Red fruit: $fruit ($count, $color)"
done
Output:
  Red fruit: apple (10, red)
  Red fruit: cherry (15, red)

Accumulate values using a subshell:
Edit 
#!/bin/sh
total=0
printf "10\n20\n30\n" | (
    while read -r n; do
        total=$((total + n))
    done
    echo "Total: $total"
)
Output:
Total: 60

Note: Variables set in piped while loops don’t persist outside due to subshell. Use a here-string or a different approach:
Edit 
#!/bin/sh
total=0
while read -r n; do
    total=$((total + n))
done <<'EOF'
10
20
30
EOF
echo "Total: $total"
Output:
Total: 60

Process with multiple passes using a subshell:
Edit 
#!/bin/sh
cat >/tmp/data.txt <<'EOF'
apple 10 red
banana 20 yellow
cherry 15 red
date 25 brown
EOF

# Pass 1: Filter
# Pass 2: Transform
cat /tmp/data.txt | grep -v "date" | awk '{print $1, $2 * 2}'
Output:
apple 20
banana 40
cherry 30

Use tee to debug pipelines:
Edit 
#!/bin/sh
echo "Debug with tee:"
printf "a\nb\nc\n" | tee /dev/stderr | tr '[:lower:]' '[:upper:]'
Output:
Debug with tee:
A
B
C
a
b
c

xargs can be used as a line processor:
Edit 
#!/bin/sh
echo "Using xargs:"
printf "file1\nfile2\nfile3\n" | xargs -I {} echo "Processing: {}"
Output:
Using xargs:
Processing: file1
Processing: file2
Processing: file3

head and tail can be used as filters:
Edit 
#!/bin/sh
echo "Head and tail:"
seq 1 10 | head -3
seq 1 10 | tail -3
Output:
Head and tail:
1
2
3
8
9
10

uniq can be used for deduplication (requires sorted input):
Edit 
#!/bin/sh
echo "Unique lines:"
printf "a\na\nb\nc\nc\nc\n" | uniq

echo "Count occurrences:"
printf "a\na\nb\nc\nc\nc\n" | uniq -c
Output:
Unique lines:
a
b
c
Count occurrences:
      2 a
      1 b
      3 c

Data can be sorted using the sort command.
Edit 
#!/bin/sh
echo "Sorted output:"
printf "cherry\napple\nbanana\n" | sort
Output:
Sorted output:
apple
banana
cherry

cut is a simple method for extracting fields from a line.
Edit 
#!/bin/sh
echo "Extract fields with cut:"
echo "name:age:city" | cut -d: -f2
Output:
Extract fields with cut:
age

Handle last line without newline:
Edit 
#!/bin/sh
echo "Handle missing final newline:"
printf "line1\nline2" | while IFS= read -r line || [ -n "$line" ]; do
    echo "  [$line]"
done
Output:
Handle missing final newline:
  [line1]
  [line2]

« Printf | Spawning Processes »

by Jose Diaz-Gonzalez | inspired by Go by Example | source