uniq

The uniq command in Unix/Linux is a utility that is used to filter out adjacent, duplicate lines from a sorted file. It is often used in conjunction with the sort command, as uniq operates on adjacent lines, identifying and removing duplicates that appear consecutively.

uniq [options] [input_file [output_file]]
Here are some common options:
    -c or --count: Prefix lines by the number of occurrences.
    -d or --repeated: Only print duplicate lines.
    -i or --ignore-case: Ignore differences in case when comparing lines.
    -u or --unique: Only print unique lines.
    -f N or --skip-fields=N: Avoid comparing the first N fields.
    -s N or --skip-chars=N: Avoid comparing the first N characters.

Examples

Display unique lines in a sorted file
sort filename | uniq
Count and display the number of occurrences for each line
sort filename | uniq -c
Display unique lines and ignore case
sort filename | uniq -i
Display only repeated lines
sort filename | uniq -d
Count and display repeated lines
sort filename | uniq -dc
Display unique lines and ignore leading whitespace
sort filename | uniq -w4
Display unique lines, count repeated lines, and ignore leading whitespace
sort filename | uniq -c -w4
Display unique lines and count repeated lines, case-insensitive
sort filename | uniq -ci
Display unique lines and count repeated lines, skip 2 characters
sort filename | uniq -c -s2
Display unique lines and count repeated lines, skip 2 fields
sort -t',' filename.csv | uniq -c -s2
Display unique lines, count repeated lines, and ignore case, skip 3 characters
sort filename | uniq -ci -s3
Display unique lines, count repeated lines, and ignore case, skip 2 fields
sort -t',' filename.csv | uniq -ci -s2
Display unique lines and count repeated lines, ignore first 2 characters
sort filename | uniq -c -s+2
Display unique lines and count repeated lines, ignore first 2 fields
sort -t',' filename.csv | uniq -c -s+2
Display unique lines, count repeated lines, and ignore case, skip 2 characters
sort filename | uniq -ci -s+2
Display unique lines, count repeated lines, and ignore case, skip 2 fields
sort -t',' filename.csv | uniq -ci -s+2
Display unique lines and count repeated lines, ignore last 2 characters
sort filename | uniq -c -s-2
Display unique lines and count repeated lines, ignore last 2 fields
sort -t',' filename.csv | uniq -c -s-2
Display unique lines, count repeated lines, and ignore case, skip 2 characters
sort filename | uniq -ci -s-2
Display unique lines, count repeated lines, and ignore case, skip 2 fields
sort -t',' filename.csv | uniq -ci -s-2
Display unique lines and count repeated lines, skip 2 bytes
sort filename | uniq -c -w2
Display unique lines and count repeated lines, skip 2 fields in a csv file
sort -t',' filename.csv | uniq -c -w2
Display unique lines and count repeated lines, ignore first 2 bytes
sort filename | uniq -c -w+2
Display unique lines and count repeated lines, ignore first 2 fields in a csv file
sort -t',' filename.csv | uniq -c -w+2
Display unique lines and count repeated lines, ignore last 2 bytes
sort filename | uniq -c -w-2
Display unique lines and count repeated lines, ignore last 2 fields in a csv file
sort -t',' filename.csv | uniq -c -w-2
Display unique lines, count repeated lines, ignore case, and skip 2 bytes
sort filename | uniq -ci -w2
Display unique lines, count repeated lines, ignore case, and skip 2 fields in a csv file
sort -t',' filename.csv | uniq -ci -w2
Display unique lines, count repeated lines, ignore case, and ignore first 2 bytes
sort filename | uniq -ci -w+2
display unique lines, count repeated lines, ignore case, and ignore first 2 fields in a csv file
sort -t',' filename.csv | uniq -ci -w+2