Combining Tools with Pipes
In this page:
A Realistic Log Analysis Pipeline
This pipeline filters log lines for errors, extracts the responsible module name, and counts how often each module appears, combining grep, awk, sort, and uniq -c into one useful report.
Example: A Realistic Log Analysis Pipeline
#!/bin/bash
cat > app.log <<'EOF'
2024-01-01 INFO auth: login ok
2024-01-01 ERROR db: connection lost
2024-01-01 ERROR auth: bad token
2024-01-01 ERROR db: timeout
2024-01-01 INFO db: reconnected
EOF
grep "ERROR" app.log | awk '{print $3}' | sort | uniq -c | sort -rn
rm -f app.log
Login to try C/C++/Java/PHP code in the editor
Building the Pipeline Incrementally
Rather than writing a five-stage pipeline in one attempt, build and verify it one stage at a time so each addition's effect is clear before moving to the next.
Example: Building the Pipeline Incrementally
#!/bin/bash
cat > data.csv <<'EOF'
name,score
alice,88
bob,42
carol,95
dave,60
EOF
echo "stage 1: skip header"
tail -n +2 data.csv
echo "stage 2: extract scores"
tail -n +2 data.csv | cut -d, -f2
echo "stage 3: sort numerically, descending"
tail -n +2 data.csv | cut -d, -f2 | sort -rn
rm -f data.csv
Login to try C/C++/Java/PHP code in the editor
Combining grep, cut, and wc
A short pipeline can answer a concrete question directly, such as "how many configuration lines set a boolean to true", without writing any custom parsing code.
Example: Combining grep, cut, and wc
#!/bin/bash
cat > settings.conf <<'EOF'
debug=true
verbose=false
cache=true
retry=true
EOF
count=$(grep "=true" settings.conf | cut -d= -f1 | wc -l)
echo "Number of enabled settings: $count"
rm -f settings.conf
Login to try C/C++/Java/PHP code in the editor
- Trying to do everything in one enormous awk or sed command instead of composing several simpler tools, making the pipeline hard to read and debug.
- Not testing each stage of a pipeline independently before chaining them, making it hard to tell which stage introduced a bug.
- Forgetting that later stages of a pipeline see only what earlier stages already transformed, not the original raw input.
- Building a pipeline incrementally, checking output after each added stage, makes debugging far easier than writing it all at once.
- grep, sed, awk, cut, sort, uniq, and wc combine naturally because they all read stdin and write stdout by default.
- A realistic pipeline often looks like: filter (grep) -> reshape (awk/cut) -> order (sort) -> summarize (uniq -c/wc).
- Formatting a pipeline across multiple lines with trailing
|improves readability for anything beyond two or three stages.
Chapter Quiz — Complete all 12 topics to unlock
0/12 topics done
Complete these topics first: