How Can I Efficiently Remove Duplicate Dictionaries from a Python List?

Removing Duplicate Dictionaries from a List in Python

When dealing with lists of dictionaries in Python, you may encounter the need to remove duplicates. Whether the duplicates share identical key-value pairs or precise key-value order, an efficient approach is necessary.

Using Hashing for Duplicate Removal

The following one-liner effectively removes duplicate dictionaries based on their key-value pairs:

[dict(t) for t in {tuple(d.items()) for d in l}]

This approach involves converting each dictionary to a tuple of its key-value pairs. Since tuples can be hashed, you can leverage the set data structure to eliminate duplicates. The resulting set contains unique tuples, which are then converted back to dictionaries.

Preserving Ordering

If the ordering of dictionaries matters, the above approach will not suffice. Instead, a more explicit method can be employed:

l = [{'a': 123, 'b': 1234},
        {'a': 3222, 'b': 1234},
        {'a': 123, 'b': 1234}]

seen = set()
new_l = []
for d in l:
    t = tuple(d.items())
    if t not in seen:
        seen.add(t)
        new_l.append(d)

print(new_l)

This code accumulates non-duplicate dictionaries in a new list while maintaining their original order. It utilizes a set to keep track of encountered tuples to ensure uniqueness.

Note:

It's important to note that in some cases, two dictionaries with identical keys and values may not result in the same tuple. This occurs when the dictionaries have different key-adding/removing sequences. If this is a concern in your scenario, consider sorting each dictionary's items before creating the tuple.

The above is the detailed content of How Can I Efficiently Remove Duplicate Dictionaries from a Python List?. For more information, please follow other related articles on the PHP Chinese website!