How to remove duplicate items from a list in Python

To remove duplicates in the list, if the order is not required, you can use list(set(original_list)), which is the fastest but does not guarantee the order; if the order is required, it is recommended to list(dict.fromkeys(original_list)), which is efficient and maintains the insertion order in Python 3.7; for dictionaries and other non-hashable types or custom deduplication logic, a loop should be used to manually control it with the seeen list or collection; if complex structures are processed, it can be converted and processed through json.dumps() or tuple(). In summary, dict.fromkeys() is generally the best choice to take into account performance, readability and order preservation.

Removing duplicate items from a list in Python is a common task, and there are several ways to do it depending on your needs—like whether you need to preserve the original order or not. Here are the most practical methods:

1. Using set() (Fastest, but doesn't preserve order)

If the order of elements doesn't matter, converting the list to a set is the quickest way to remove duplicates, since sets only store unique values.

 original_list = [3, 1, 2, 2, 1, 3, 4]
unique_list = list(set(original_list))
print(unique_list) # Output might be: [1, 2, 3, 4]

⚠️ Note: The order of items is not guaranteed because sets are unordered collections.

This method is efficient for large lists when order isn't important.

2. Using dict.fromkeys() (Preserves order and is efficient)

Starting from Python 3.7, dictionaries preserve insertion order. You can use dict.fromkeys() to remove duplicates while keeping the original order.

 original_list = [3, 1, 2, 2, 1, 3, 4]
unique_list = list(dict.fromkeys(original_list))
print(unique_list) # Output: [3, 1, 2, 4]

This is generally the best approach when you want to keep the first occurrence of each item and maintain order.

• It's fast (similar performance to using a set)
• Preserves order
• Handles unhashable types? No — only works with hashable items (like numbers, strings, tuples)

3. Using a loop for complex cases (Maximum control)

If you're dealing with unhashable types like dictionaries or lists, or need custom logic to determine duplicates, use a loop.

 original_list = [{'id': 1}, {'id': 2}, {'id': 1}]
unique_list = []
seen = []

for item in original_list:
    if item['id'] not in seen:
        see.append(item['id'])
        unique_list.append(item)

print(unique_list) # Output: [{'id': 1}, {'id': 2}]

This method is slower for large data but gives you full control. You can define what makes items "duplicate" (eg, based on a key).

Alternatively, for unhashable types, you might use a more advanced approach with json.dumps() or tuple() conversion if structure allows.

4. Using list comprehension with a helper set (Order preserved, good performance)

You can manually track see items using a set during iteration:

 original_list = [3, 1, 2, 2, 1, 3, 4]
see = set()
unique_list = []

for item in original_list:
    if item not in see:
        seen.add(item)
        unique_list.append(item)

print(unique_list) # Output: [3, 1, 2, 4]

This is slightly more verbose than dict.fromkeys() , but useful when you need to apply extra logic during filtering.

Summary: Which method to choose?

• ✅ Preserve order & simple data? → list(dict.fromkeys(my_list))
• ✅ Don't care about order? → list(set(my_list))
• ✅ Dealing with dictionaries or custom logic? → Use a loop with a tracking variable
• ⚠️ List contains lists or unhashable items? → You'll need to convert or compare manually

Most of the time, dict.fromkeys() is the sweet spot for readingability, speed, and order preservation.

Basically, pick the method that fits your data and requirements.

The above is the detailed content of How to remove duplicate items from a list in Python. For more information, please follow other related articles on the PHP Chinese website!