Python deduplicates values ​​in the list structure

I encountered a problem today. I used the itertools.groupby function after a casual prompt from a colleague. However, this thing was not used in the end.

The problem is to deduplicate the news IDs in a list, and ensure that the order remains unchanged after deduplication.
Intuitive method
The simplest idea is:

ids = [1,2,3,3,4,2,3,4,5,6,1]
news_ids = []
for id in ids:
  if id not in news_ids:
    news_ids.append(id)

print news_ids

This is also the case It works, but it doesn't look cool enough.
Use set
Another solution is to use set:

ids = [1,4,3,3,4,2,3,4,5,6,1]
ids = list(set(ids))

Sort again by index
Finally solved in this way:

ids = [1,4,3,3,4,2,3,4,5,6,1]
news_ids = list(set(ids))
news_ids.sort(ids.index)

Use itertools .grouby
The article mentioned itertools.grouby at the beginning. If you don’t care about the order of the list, you can use this:

##

ids = [1,4,3,3,4,2,3,4,5,6,1]
ids.sort()
it = itertools.groupby(ids)

for k, g in it:
  print k

About the principle of itertools.groupby, you can see here: //m.sbmmt.com/

Use reduce
Netizen reatlk left a message for another solution. I add and explain here:

In [5]: ids = [1,4,3,3,4,2,3,4,5,6,1]

In [6]: func = lambda x,y:x if y in x else x + [y]

In [7]: reduce(func, [[], ] + ids)
Out[7]: [1, 4, 3, 2, 5, 6]

The above is the code I run in ipython, in which lambda x, y:x if y in x else x + [y] is equivalent to lambda x,y: y in x and x or x+[y] .

More Python performs operations on values ​​in the list structure For articles related to deduplication, please pay attention to the PHP Chinese website!