Detailed examples of several ideas for deduplication of JavaScript arrays

The demand for data deduplication is actually that tool libraries such as lodash have mature and complete implementations and can be maturely used in production environments. But this does not prevent us from the perspective of thinking expansion to see how duplication removal can be achieved using several ideas. This article mainly shares with you several ideas for deduplication of JavaScript arrays.

The first is the implementation of the conventional double-layer circular comparison idea

function doubleLoopUniq(arr) {
  let result = [];
  for (let i = 0, len = arr.length, isExist; i <p>Use the built-in indexOf of js to remove duplicates</p><pre class="brush:php;toolbar:false">function indexOfUniq(arr) {
  let result = [];
  for (let i = 0, len = arr.length; i <p>After sorting, compare before and after to remove duplicates</p> <pre class="brush:php;toolbar:false">function sortUniq(arr) {
  let result = [], last;
  // 这里解构是为了不对原数组产生副作用
  [ ...arr ].sort().forEach(item => {
    if (item != last) {
      result.push(item);
      last = item;
    }
  });
  return result;
}

Deduplication through hashTable

function hashUniq(arr) {
  let hashTable = arr.reduce((result, curr, index, array) => {
    result[curr] = true;
    return result;
  }, {})
  return Object.keys(hashTable).map(item => parseInt(item, 10));
}

ES6 SET one line of code to achieve deduplication

function toSetUniq(arr) {
  return Array.from(new Set(arr));
}

splice deduplication (directly operates the array itself, with side effects)

function inPlaceUniq(arr) {
  let idx = 0;
  while (idx <p>Finally in Run a simple test under nodejs to see which one is more efficient~</p><pre class="brush:php;toolbar:false">let data = [];
for (var i = 0; i <p>The results are as follows</p><pre class="brush:php;toolbar:false">hashTable 168ms
sortUniq 332ms
toSetUniq 80ms
indexOfUniq 4280ms
doubleLoopUniq 13303ms
inPlaceUniq 9977ms

Extended thinking: If the elements in the array are objects, how to remove duplicates?

Since it is a reference type, deepEqual will inevitably be used. Although this idea can answer this problem, it is inevitably not efficient enough.

It can also be seen from the above test that deduplication through new Set and hashTable is the most efficient.
So there is no doubt that we need to transform based on these two methods. I want to use hashTable.
On the other hand, in order to reduce the time-consuming caused by deep comparison, I try to use JSON.stringify to reference The type is converted to a basic type.

function collectionUniq(collection) {
  let hashTable = {};
  collection.forEach(item => {
    hashTable[JSON.stringify(item)] = true;
  })
  return Object.keys(hashTable).map(item => JSON.parse(item))
}

Then here comes the problem. We all know that the attributes of objects are unordered. If the data is like this, then it is GG.

let collection = [ { a: 1, b: 2, c: 3 }, { b: 2, c: 3, a: 1 } ]

There is a toHash idea, After a basic deduplication of this array, in order to ensure accuracy,
first traverse the JSON string=>
Pass charCodeAt() gets the unicode encoding of each string =>
and adds them to get a total number. Finally, compare them in pairs. Those with equal values ​​are duplicates, thus achieving the effect of deduplication.

function toHash(obj) {
  let power = 1;
  let res = 0;
  const string = JSON.stringify(obj, null, 2);
  for (let i = 0, l = string.length; i <p>This is just a basic idea for implementation, and there is a lot of room for improvement. In order to reduce the possibility of hash collision, the weight of some special characters can be increased or decreased. </p><p><strong>The key point is to ensure that the probability of collision is smaller than winning the jackpot. </strong></p><p class="comments-box-content">Related recommendations: <br></p><p class="comments-box-content"><a href="//m.sbmmt.com/js-tutorial-386521.html" target="_self"> Sharing several methods of deduplication in JavaScript arrays</a></p><p class="comments-box-content"><a href="//m.sbmmt.com/php-weizijiaocheng-386230.html" target="_self"> Implementing array deduplication in PHP Method code</a></p><p class="comments-box-content"><a href="//m.sbmmt.com/js-tutorial-384672.html" target="_self">JS simple implementation of array deduplication method analysis</a></p>

The above is the detailed content of Detailed examples of several ideas for deduplication of JavaScript arrays. For more information, please follow other related articles on the PHP Chinese website!