uniq command in Linux with examples

The Linux uniq command is used to remove all repeated lines in a file. This command is used when a line is repeated multiple times and replaces these multiple lines with one line. This command is designed to work on sorted files.

Here, we’ll explain how the uniq command works, demonstrate various options you can use and provide practical examples to help you understand its capabilities.

What is the Purpose of the uniq Command?

Remove repetitious lines from the sorted 'input-file' and send unique lines to the 'output-file'. If no 'output-file' is specified, the output of the command is sent to standard output. if no 'input-file' is specified, the command takes input from standard output.

Syntax

$ uniq [option] [input-file] [output-file]

• input-file: The file containing lines you want to process.
• output-file: (Optional) The file to which you want to save the result. If no output file is provided, the result is printed on the terminal.

Key Options for the uniq Command

uniq command Examples in Linux

Create a sample input file.

$ cat sample

Input: This is a test file for the uniq command. It contains some repeated lines. It contains some repeated lines. And some are different. It contains some repeated lines. It contains some repeated lines.

1. Removing duplicate lines when file is not sorted.

$ uniq sample

Output:

This is a test file for the uniq command. It contains some repeated lines. And some are different. It contains some repeated lines.

2. Removing duplicate files after sorting.

$ sort sample -> sample1
$ uniq sample1

Output:

And some are different. It contains some repeated lines. This is a test file for the uniq command.

3. Finding number of times the lines are repeated.

$ uniq -c sample1

Output:

1 And some are different.
4 It contains some repeated lines.
1 This is a test file for the uniq command.

4. Finding repeated lines.

$ uniq -d sample1

Output:

It contains some repeated lines.

5. Finding lines which are unique and store in another file.

$ uniq -u sample1 out
$ cat out

Output:

And some are different.
This is a test file for the uniq command.

Conclusion

The uniq command in Linux is an invaluable tool for anyone working with text files that may contain duplicate lines. By using the available options, you can easily filter, count, or isolate repeated and unique lines, making it a versatile command for a variety of file management and data processing tasks.