The uniq command in Linux is used to filter out or report repeated lines in a file or input. It is often used in conjunction with the sort command, as uniq only identifies duplicates that are adjacent. This means you typically want to sort the data first if you're looking to remove all duplicates from an unsorted file.

Basic Syntax

uniq [options] [input_file] [output_file]

Common Options

• -c : Counts the number of occurrences of each line.
• -d : Displays only duplicate lines.
• -u : Displays only unique lines.
• -i : Ignores case differences when comparing lines.

Examples with Output

Example 1: Basic Usage

Suppose you have a file named file.txt that contains the following lines:

apple
banana
apple
cherry
banana

Command

uniq file.txt

Output

apple
banana
apple
cherry
banana

Explanation:

• Since the lines are not adjacent, uniq does not remove any duplicates in this case.

Example 2: Using uniq After sort

To remove all duplicates regardless of their positions, first sort the file.

Command

sort file.txt | uniq

Output

apple
banana
cherry

Explanation:

• The sort command organizes the lines, placing identical lines next to each other. Then, uniq removes duplicates, leaving only one instance of each line.

Example 3: Counting Occurrences with -c

To count how many times each line appears, use the -c option.

Command

sort file.txt | uniq -c

Output

      2 apple
      2 banana
      1 cherry

Explanation:

• Each line is preceded by a count of occurrences. In this case, apple and banana both appear twice, while cherry appears once.

Example 4: Displaying Only Duplicates with -d

To show only the lines that appear more than once, use the -d option.

Command

sort file.txt | uniq -d

Output

apple
banana

Explanation:

• This command lists only the lines that are duplicated, so only apple and banana are shown.

Example 5: Displaying Only Unique Lines with -u

To show only the lines that are unique (not repeated), use the -u option.

Command

sort file.txt | uniq -u

Output

cherry

Explanation:

• The output shows only lines that appear once, so cherry is the only unique line.

Example 6: Ignoring Case Differences with -i

To ignore case when determining duplicates, use the -i option.

Suppose file2.txt contains:

Apple
banana
apple
Cherry
BANANA

Command

sort file2.txt | uniq -i

Output

Apple
banana

Explanation:

• The command ignores the case of apple and Apple, as well as banana and BANANA, resulting in only one instance of each unique line being shown.

Summary of Common uniq Commands

The uniq command is useful for processing text data, especially when working with sorted input to manage duplicates efficiently.