🛠️Dev Toolbelt

Lesson 48 of 117

25 min

Power Tools

grep, rg, sed, awk, find, fzf, jq, ssh.

The shell's real magic isn't any one command. It's gluing small tools together with pipes to build a one-liner that would take 50 lines in Python. This lesson is the toolbelt: searching, filtering, transforming, and reaching into remote machines.

grep and ripgrep: find text

grep searches files for lines matching a pattern. ripgrep (the rg command) is the modern, faster, smarter cousin that respects .gitignore by default. If you can install one tool today, install ripgrep.

bash

# classic grep
grep "TODO" src/app.js
grep -r "TODO" src/          # recursive
grep -in "error" server.log  # -i ignore case, -n line numbers

# ripgrep (faster, smarter defaults)
rg "TODO"                    # recursive by default, skips node_modules
rg "useState" -t ts          # only TypeScript files
rg "fetch\(" -A 3           # show 3 lines after each match
rg -l "TODO"                 # list filenames only, no matches

bash

rg "useState" src/
# src/components/Counter.tsx
# 3:import { useState } from "react";
# 7:  const [count, setCount] = useState(0);
#
# src/hooks/useForm.ts
# 12:  const [values, setValues] = useState(initial);

sed: quick stream edits

sed reads a stream and applies edits. 90% of real-world use is just substitution: s/find/replace/.

bash

# replace first occurrence per line
echo "hello world" | sed 's/world/there/'
# hello there

# replace all (the g flag)
echo "a a a" | sed 's/a/b/g'
# b b b

# edit a file in place (macOS needs the empty string after -i)
sed -i '' 's/localhost:3000/api.example.com/g' config.js

# delete lines matching a pattern
sed '/^#/d' config.txt   # remove all lines starting with #

sed -i is permanent

In-place editing has no undo. Commit your changes to git first, or pipe to a new file until you trust the command.

awk: when your data is tabular

awksplits each line into fields and lets you do simple math or filtering. It's ancient and weird, but for tabular output it's unbeatable.

bash

ps aux | awk '{print $2, $11}'
# PID  COMMAND
# 1    /sbin/launchd
# 432  /usr/bin/ssh-agent
# 9821 node

# sum a column
du -k * | awk '{total += $1} END {print total/1024 " MB"}'
# 248.4 MB

# filter then format
df -h | awk '$5+0 > 80 {print $1, $5}'   # show partitions over 80% full

find: locate files by metadata

bash

find . -name "*.log"                       # all .log files anywhere below
find . -type d -name "node_modules"        # directories named node_modules
find . -size +100M                         # files larger than 100MB
find . -mtime -7                           # modified in the last 7 days
find . -name "*.tmp" -delete               # find and delete (be careful)

# combined with another command via -exec
find . -name "*.ts" -exec wc -l {} +

rg --files is often faster

For pure file listing in a project, rg --files respects .gitignore and runs in milliseconds where find can crawl for ages.

jq: surgical JSON

jq is grep for JSON. Filter, extract, reshape, all from the command line. You will use this constantly when poking at HTTP APIs.

bash

curl -s https://api.github.com/repos/microsoft/vscode | jq '.stargazers_count'
# 162847

# extract specific fields
curl -s https://api.github.com/users/torvalds/repos \
  | jq '.[] | {name, stars: .stargazers_count}'
# { "name": "linux", "stars": 178432 }
# { "name": "subsurface", "stars": 2103 }
# ...

# pretty-print a local file
cat package.json | jq

# filter an array
jq '.dependencies | keys' package.json
# [ "next", "react", "react-dom" ]

Pipes and redirection

This is the glue. A pipe |feeds one command's output into the next. Redirections (>, >>, 2>&1) send output to files instead of the screen.

bash

# pipes
cat access.log | grep "POST" | wc -l            # count POST requests
ps aux | grep node | awk '{print $2}'           # PIDs of all node processes

# redirect stdout to a file (overwrite)
npm run build > build.log

# append instead of overwrite
echo "done" >> build.log

# stderr is a separate stream
npm run test 2> errors.log                       # only errors go to file
npm run test > all.log 2>&1                      # combine stdout + stderr
npm run test &> all.log                          # shorthand in bash/zsh

# silence stderr entirely
some-noisy-command 2> /dev/null

Streams: 0, 1, 2

Every process has three default streams: stdin (0), stdout (1), stderr (2). Pipes pass stdout. Redirections can target any of them. 2>&1 means "redirect stderr to wherever stdout is currently going."

Background jobs

Stick an & on the end of a command and the shell runs it in the background, immediately giving you back the prompt.

bash

npm run dev &
# [1] 51932

jobs
# [1]+  Running    npm run dev &

# bring it back to the foreground
fg %1

# suspend a running foreground job with Ctrl-Z, then push it to background
# (Ctrl-Z)
bg %1

# kill a background job
kill %1

For long-running things (servers, log tails, build watchers) most devs actually prefer a terminal multiplexer like tmux or just opening a second tab. But knowing & and jobs is essential.

SSH: your remote shell

sshopens a shell on a remote computer. The connection is encrypted. It's how you log into servers, deploy code, and run commands on the cloud.

bash

ssh ada@server.example.com
# (you're now on the server)

# run one command and exit
ssh ada@server "df -h"

# specify a port
ssh -p 2222 ada@server.example.com

# copy files over ssh
scp local-file.txt ada@server:/var/www/
rsync -avz ./build/ ada@server:/var/www/site/

ssh-keygen and ssh config

Typing your password every connection is annoying and less secure than keys. Generate a key once, copy the public half to the server, and you're passwordless.

bash

# generate a modern Ed25519 key
ssh-keygen -t ed25519 -C "ada@laptop"
# saves ~/.ssh/id_ed25519 (private) and id_ed25519.pub (public)

# install your public key on the server
ssh-copy-id ada@server.example.com

# now this works without a password
ssh ada@server.example.com

Even better: an ~/.ssh/config file gives connections nice nicknames and remembers all your flags.

~/.ssh/config

Host prod
  HostName server.example.com
  User ada
  Port 2222
  IdentityFile ~/.ssh/id_ed25519

Host github.com
  User git
  IdentityFile ~/.ssh/id_ed25519_github

bash

ssh prod                # uses everything from the config above
scp deploy.sh prod:/tmp/

Never share your private key

The .pub file is the one you copy to servers. The non-.pub file stays on your machine, period. Leaking a private key is like leaving your house keys in a public park.

A real-world pipeline

Imagine your server logs have lines like [2024-05-24T14:02:13] 200 GET /api/users 18ms. Show me the 10 slowest endpoints from the last hour:

bash

tail -n 100000 access.log \
  | rg "$(date -v-1H +%Y-%m-%dT%H)" \
  | awk '{print $5, $4}' \
  | sort -rn \
  | head -10
# 1842ms GET /api/reports/heavy
# 1320ms POST /api/upload
# 980ms  GET /api/users
# ...

That's 5 small tools doing one thing each, pipelined. This is the terminal's real power: you assemble it.

Quick quiz

Quiz1 / 4

What does 2>&1 mean?

Recap

grep / ripgrep find lines. sed rewrites them. awk handles columns. find locates files by metadata. jq surgically reshapes JSON.
Pipes (|) chain tools. Redirections (>, >>, 2>&1) route output to files.
& and jobs manage background processes; fg/bg move them between foreground and background.
ssh opens a shell on a remote server. ssh-keygen + ~/.ssh/config make it passwordless and ergonomic.