note.nkmk.me

Check If a List Has Duplicates in Python

Modified: | Tags: Python, List

This article explains how to check if a Python list has duplicates, or in other words, if all elements in the list are unique.

Contents

See the following article for removing or extracting duplicates from a list.

Check if a list has duplicates (when no unhashable objects are present)

Use set() if a list does not contain unhashable objects like other lists.

When a list is passed to set(), it returns a set, which ignores duplicates and keeps only unique elements.

You can compare the number of elements in this set and the original list using the built-in function len(). If the number of elements matches, it suggests that the original list had no duplicates. If the counts differ, the original list contains duplicates.

The function below returns False if there are no duplicates, and True if duplicates exist:

def has_duplicates(seq):
    return len(seq) != len(set(seq))

l = [0, 1, 2]
print(has_duplicates(l))
# False

l = [0, 1, 1, 2]
print(has_duplicates(l))
# True
source: list_duplicate_check.py

While the sample code above uses a list, the function can also be applied to a tuple.

Note that a set cannot include unhashable objects like lists. Therefore, trying to convert a list that contains other lists (two-dimensional lists or lists of lists) to a set will cause a TypeError.

l_2d = [[0, 1], [1, 1], [0, 1], [1, 0]]
# print(has_duplicates(l_2d))
# TypeError: unhashable type: 'list'
source: list_duplicate_check.py

Check if a list has duplicates (when unhashable objects are present)

For lists that include other lists, the following function can check for duplicates:

def has_duplicates2(seq):
    seen = []
    unique_list = [x for x in seq if x not in seen and not seen.append(x)]
    return len(seq) != len(unique_list)

l_2d = [[0, 0], [0, 1], [1, 1], [1, 0]]
print(has_duplicates2(l_2d))
# False

l_2d = [[0, 0], [0, 1], [1, 1], [1, 1]]
print(has_duplicates2(l_2d))
# True
source: list_duplicate_check.py

Instead of set(), this function generates a list of only unique values using list comprehension and then compares element counts. Read the following article for more details.

This function is also compatible with lists that only contain hashable objects.

l = [0, 1, 2]
print(has_duplicates2(l))
# False

l = [0, 1, 1, 2]
print(has_duplicates2(l))
# True
source: list_duplicate_check.py

The above example checks if the list contains identical sublists. You can also identify duplicate elements within each sublist by flattening the original list to a single dimension.

l_2d = [[0, 1], [2, 3]]
print(sum(l_2d, []))
# [0, 1, 2, 3]

print(has_duplicates(sum(l_2d, [])))
# False

l_2d = [[0, 1], [2, 0]]
print(has_duplicates(sum(l_2d, [])))
# True
source: list_duplicate_check.py

The example uses sum() to flatten the list. However, you can also achieve the same result using itertools.chain.from_iterable(). To flatten a list with more than three dimensions, define a new function. See the following article for more information.

Related Categories

Related Articles