How to Remove Duplicate Lines in Linux

There are 2 options to remove duplicate lines in some file.

Sample file:

aaaaa
ddddd
bbbbb
aaaaa
fffff
eeeee
aaaaa
eeeee
ggggg
fffff
  1. Using sort, with option -u or --unique. The output will be sorted.
    sort -u sample.txt

    Output:

    aaaaa
    bbbbb
    ddddd
    eeeee
    fffff
    ggggg
  2. Using awk. With this, the order of the lines is not changed (no sorting).
    awk '!x[$0]++' sample.txt

    Output:

    aaaaa
    ddddd
    bbbbb
    fffff
    eeeee
    ggggg

To save into a new file, just add “> newfile.txt” (without quotes):

awk '!x[$0]++' sample.txt > newfile.txt

Hope it helps. 🙂