The Python list deduplication method you should know

Preface

List deduplication is a common problem when writing Python scripts, because no matter where the source data comes from, when we convert it into a list, there may be expected The result is not our final result. The most common thing is that there are duplicate elements in the list. At this time, the first thing we need to do is deduplication.

Let’s take the simplest way first, using Python’s built-in data type set.

Suppose our list data is like this:

level_names = [
 u'Second Level',
 u'Second Level',
 u'Second Level',
 u'First Level',
 u'First Level'
]

Because the elements of the set cannot be repeated, so When converting a list into a set, duplicate elements will be automatically removed. This is the basic principle. The code is as follows:

>>> the_list = set(level_names)
>>> print(the_list)
set([u'Second Level', u'First Level'])

The disadvantage of this method is that it is converted into a list again. It is impossible to save the previous list order. If there is no such requirement, this method is the simplest answer. Maybe some friends think it is too simple. There is no technical content in this. Yes, so in general interview questions, you are asked to list. The heavy words are usually written like this:

Please write down the method to remove the heavy words from the list (set cannot be used)

People have stated that set cannot be used. , so, sometimes this trick doesn’t work, so of course it doesn’t trouble us, we have other methods.

We all know that lists can be traversed, and the problem of traversing is simple. We then define an empty list, then traverse the list with data, and add a judgment when traversing. If it is not in the empty list, add Enter it and throw it away if you have it. The code is as follows:

the_list = []
for level in level_names:
 if level not in the_list:
  the_list.append(level)
print(the_list)

Do you think this method is okay, but this method is suitable for ordinary small lists. No problem, but if you encounter a super large list, you will not be able to do what you want, because the list in the_list becomes very large, which will affect the efficiency when making judgments, because the list is searched in index order, and when the amount of data is large, it will Slow down.

Maybe you want to ask, what should I do if I encounter a large list? Is there a more awesome way? Of course, let's continue. Since using lists when judging will affect efficiency, let's change our thinking. We use sets. Then you may ask, are sets faster? Yes, because the hash function used by set When searching for values, although the set is unordered, the position is fixed. It only takes one time to check whether a specific element exists. Someone on the Internet did a comparison of searching for elements in lists and sets. Under the same data conditions, it took 16 minutes to use list. , it takes 52 seconds to use set. You can see the effect from this comparison. Not much else to say, just post the code:

the_list = []
the_set = set()
for level in level_names:
 if level not in the_set:
  the_set.add(level)
  the_list.append(level)
print(the_list)

More you For articles related to Python list deduplication methods that you should know, please pay attention to the PHP Chinese website!