sed

sed which stands for stream editor is a unix utility that was developed by Lee E. McMahon at Bell Labs in the 70s. It is a powerful text processing utility.

Syntax: 

sed 's/original-string/replacement-string/[flags]' [input-file]

Examples: 

# to remove everything from a line before a word
sed 's/.*WORD//'

# remove everything from a line after a character or pattern
sed 's/WORD.*//'

# extract everything between two words
sed 's/.*START//; s/END.*//'

# clean up white space in the front of the output globally
sed 's/^[[:space:]]*//g'

# clean up white space in the end of the output globally
sed 's/[[:space:]]*$//g'

# clean up white space in the the front and in the end of the output globally
sed 's/^[[:space:]]*//g; s/[[:space:]]*$//'

# extract everything between two words and clean up white space
sed 's/.*START//; s/END.*//; s/^[[:space:]]*//g; s/[[:space:]]*$//'

# replace all null in the file with |
sed 's/null/|/g' change.csv > output.csv

Some more examples:

Replace text in a file
sed 's/old_text/new_text/' filename
Replace text in a specific line
sed '5s/old_text/new_text/' filename
Replace text in all lines containing a pattern
sed '/pattern/s/old_text/new_text/' filename
Replace text only in lines matching a pattern
sed '/pattern/ s/old_text/new_text/' filename
Delete lines containing a pattern
sed '/pattern/d' filename
Delete empty lines
sed '/^$/d' filename
Add text at the beginning of each line
sed 's/^/prefix/' filename
Add text at the end of each line
sed 's/$/suffix/' filename
Print only specific lines
sed -n '5p;10p' filename
Substitute using a delimiter
sed 's|old_text|new_text|' filename
Replace text in a range of lines
sed '3,7s/old_text/new_text/' filename
Remove leading whitespace
sed 's/^[ \t]*//' filename
Substitute only the first occurrence in each line
sed 's/old_text/new_text/' filename
Insert a line before a matching pattern
sed '/pattern/i\inserted_line' filename
Append a line after a matching pattern
sed '/pattern/a\appended_line' filename
Remove html tags
sed 's/<[^>]*>//g' filename.html
Uppercase first letter of each word
sed 's/\b\(.\)/\u\1/g' filename
Replace a range of lines with external content
sed '10,20r file_to_insert.txt' filename
Reverse the order of lines
sed '1!G;h;$!d' filename
Remove trailing whitespace
sed 's/[ \t]*$//' filename

Command Line Options

Option Description Example
-n Suppress default pattern space printing sed -n '3 p' test.txt
-i Backup and modify input file directly sed -ibak 's/John/Johnny/' test.csv
-f Execute sed script file sed -f sedscript.sed test.txt
-e Execute multiple sed commands sed -e 'command1' -e 'command2' -e 'command3' file.txt

Sed substitute command and flags

Flag Description Example
g Global substitution sed 's/search/replace/g' file.txt
1,2... Substitute the nth occurrence sed 's/search/replace/2' file.txt
p Print only the substituted line sed -n 's/search/replace/p' file.txt
w Write only the substituted line to a file sed -n 's/search/replace/w out.txt' file.txt
i Ignore case while searching sed 's/search/replace/i' file.txt
e Substitute and execute in the command line sed -e 's/^/ls -l /' file.txt
~/|^@! Substitution delimiter can be any character sed 's@/usr/local/bin@/usr/bin@' file.txt
& Gets the matched pattern to use for replacement sed 's/^.*/<&>/' file.txt #Encloses whole line between < and >
\( \) Group using \( and \). Use \1, \2 for replacement sed 's/\([^,]*\),\([^,]*\),\([^,]*\).*/\1,\2/g' file.txt

Sed commands

Command Description Example
p Print Pattern space sed -n '1,5 p' file.txt
d Delete lines sed -n '1,5 d' file.txt
w Write pattern space to file sed -n '1,5 w output.txt' file.txt
a Append line after sed '4 a replacement-line' file.txt
i Insert line before sed '5 i replacement-line' file.txt
c Change line sed '10 c replacement-line' file.txt
l Print hidden characters sed -n l file.txt
= Print line numbers sed = file.txt | sed '{N;s/\n/ /}'
y Change case sed 'y/abcde/ABCDE/' file.txt
q Quit sed sed '6 q' file.txt
r Read from file sed '$ r sample.txt' file.txt
# Comment inside sed script #

Sed hold and pattern space commands

Command Description
n Print pattern space, empty pattern space, and read next line.
x Swap pattern space with hold space
h Copy pattern space to hold space
H Append pattern space to hold space
g Copy hold space to pattern space
G Append hold space to pattern space

Loops and multi-line sed commands

Command Description
b label Branch to a label (for looping)
t label Branch to a label only on successful substitution (for looping)
:label Label for the b and t commands (for looping)
N Append next line to pattern space
P Print 1st line in multi-line
D Delete 1st line in multi-line 
References:
(n.d.). Retrieved from https://www.gnu.org/software/sed/manual/sed.txt