{"id":2873,"date":"2020-01-29T06:19:55","date_gmt":"2020-01-29T06:19:55","guid":{"rendered":"https:\/\/www.askpython.com\/?p=2873"},"modified":"2022-08-06T13:17:36","modified_gmt":"2022-08-06T13:17:36","slug":"remove-duplicate-elements-from-list-python","status":"publish","type":"post","link":"https:\/\/www.askpython.com\/python\/remove-duplicate-elements-from-list-python","title":{"rendered":"Remove Duplicate Elements from List in Python"},"content":{"rendered":"\n

In this article, we’ll look at how we can remove duplicate elements from List in Python<\/a>. There are multiple ways of approaching this problem, and we will show you some of them.<\/p>\n\n\n\n

\n\n\n\n

Methods to Remove Duplicate Elements from List – Python<\/h2>\n\n\n\n

1. Using iteration<\/h2>\n\n\n\n

To remove duplicate elements from List in Python, we can manually iterate through the list and add an element to the new list if it is not present. Otherwise, we skip that element.<\/p>\n\n\n\n

The code is shown below:<\/p>\n\n\n

\na = [2, 3, 3, 2, 5, 4, 4, 6]\n\nb = []\n\nfor i in a:\n    # Add to the new list\n    # only if not present\n    if i not in b:\n        b.append(i)\n\nprint(b)\n<\/pre><\/div>\n\n\n
Output<\/strong><\/p>\n\n\n
\n[2, 3, 5, 4, 6]\n<\/pre><\/div>\n\n\n
The same code can be written using List Comprehension to reduce the number of lines of code, although it is essentially the same as before.<\/p>\n\n\n
\na = [2 3, 4, 2, 5, 4, 4, 6]\nb = []\n[b.append(i) for i in a if i not in b]\nprint(b)\n<\/pre><\/div>\n\n\n
The problem with this approach is that it is a bit slow since a comparison is done for every element in the new list, while already iterating through our original list. <\/p>\n\n\n\n
This is computationally expensive, and we have other methods to deal with this issue. You should use this only if the list size is not very large. Otherwise, refer to the other methods.<\/p>\n\n\n\n
2. Using set()<\/h2>\n\n\n\n
A simple and fast approach to remove duplicate elements from list in Python would be to use Python’s built-in set()<\/code> method to convert the list elements into a unique set, following which we can convert it into a List now removed of all its duplicate elements.<\/p>\n\n\n
\nfirst_list = [1, 2, 2, 3, 3, 3, 4, 5, 5, 6]\n\n# Convert to a set first\nset_list = set(first_list)\n\n# Now convert the set into a List\nprint(list(set_list))\n\nsecond_list = [2, 3, 3, 2, 5, 4, 4, 6]\n\n# Does the same as above, in a single line\nprint(list(set(second_list)))\n<\/pre><\/div>\n\n\n
Output<\/strong><\/p>\n\n\n
\n[1, 2, 3, 4, 5, 6]\n[2, 3, 4, 5, 6]\n<\/pre><\/div>\n\n\n
The problem with this approach is that the original List order is not maintained as with the case of the second List since we create the new List from an unordered Set. so if you wish to still preserve the relative ordering, you must avoid this method.<\/p>\n\n\n\n
3. Preserving Order: Use OrderedDict<\/h2>\n\n\n\n
If you want to preserve the order while you remove duplicate elements from List in Python, you can use the OrderedDict<\/strong> class from the collections<\/strong> module.<\/p>\n\n\n\n
More specifically, we can use OrderedDict.fromkeys(list)<\/code> to obtain a dictionary having duplicate elements removed, while still maintaining order. We can then easily convert it into a list<\/a> using the list()<\/code> method.<\/p>\n\n\n
\nfrom collections import OrderedDict\n\na = [2, 3, 3, 2, 5, 4, 4, 6]\n\nb = list(OrderedDict.fromkeys(a))\n\nprint(b)\n<\/pre><\/div>\n\n\n
Output<\/strong><\/p>\n\n\n
\n[2, 3, 5, 4, 6]\n<\/pre><\/div>\n\n\n
\n
NOTE<\/strong>: If you have Python 3.7<\/strong> or later, we can use the built in dict.fromkeys(list)<\/code> instead. This will also guarantee the order.<\/p>\n<\/div><\/div>\n\n\n\n
As you can observe, the order is indeed maintained, so we get the same output as of the first method. But this is much faster! This is the recommended solution to this problem. But for illustration, we will show you a couple of more approaches to remove duplicate elements from List in Python.<\/p>\n\n\n\n
4. Using list.count()<\/h2>\n\n\n\n
The list.count()<\/code> method returns the number of occurrences of the value. We can use it along with the remove()<\/code> method to eliminate any duplicate elements. But again, this does not<\/strong> preserve the order.<\/p>\n\n\n\n
Note that this method modifies the input list in place, so the changes are reflected there itself.<\/p>\n\n\n
\na = [0, 1, 2, 3, 4, 1, 2, 3, 5]\n\nfor i in a:\n    if a.count(i) > 1:\n        a.remove(i)\n\nprint(a)\n<\/pre><\/div>\n\n\n
Output<\/strong><\/p>\n\n\n
\n[0, 4, 1, 2, 3, 5]\n<\/pre><\/div>\n\n\n
Everything seems fine, isn’t it?<\/strong> <\/p>\n\n\n\n
But, there is a small issue with the above code. <\/p>\n\n\n\n
When we are iterating over the list using the for loop and removing the element at the same time, the iterator skips one element. So, the code output depends on the list elements and if you are lucky then you will never get the issue. Let’s understand this scenario with a simple code.<\/p>\n\n\n
\na = [1, 2, 3, 2, 5]\n\nfor i in a:\n    if a.count(i) > 1:\n        a.remove(i)\n    print(a, i)\n\nprint(a)\n<\/pre><\/div>\n\n\n
Output<\/strong>:<\/p>\n\n\n
\n[1, 2, 3, 2, 5] 1\n[1, 3, 2, 5] 2\n[1, 3, 2, 5] 2\n[1, 3, 2, 5] 5\n[1, 3, 2, 5]\n<\/pre><\/div>\n\n\n
You can see that the for loop is executed only four times and it’s skipping 3, the next element after the remove() call. If you pass the input list as [1, 1, 1, 1], the final list will be [1, 1].<\/p>\n\n\n\n
So, is there any workaround?<\/strong><\/p>\n\n\n\n
Of course, there is a workaround. Use the copy of the list in the for loop but remove the elements from the main list. A simple way to create a copy of the list is through slicing. Here is the update code that will work fine in all the cases.<\/p>\n\n\n
\na = [1, 1, 1, 1]\n\nfor i in a[:]:  # using list copy for iteration\n    if a.count(i) > 1:\n        a.remove(i)\n    print(a, i)\n\nprint(a)\n<\/pre><\/div>\n\n\n
Output:<\/p>\n\n\n
\n[1, 1, 1] 1\n[1, 1] 1\n[1] 1\n[1] 1\n[1]\n<\/pre><\/div>\n\n\n
5. Using sort()<\/h2>\n\n\n\n
We can use the sort()<\/code> method to sort the set that we obtained in approach 2. This will also remove any duplicates, while preserving the order, but is slower than the dict.fromkeys()<\/code> approach.<\/p>\n\n\n
\na = [0, 1, 2, 3, 4, 1, 2, 3, 5]\nb = list(set(a))\nb.sort(key=a.index)\nprint(b)   \n<\/pre><\/div>\n\n\n
Output<\/strong><\/p>\n\n\n
\n[0, 1, 2, 3, 4, 5]\n<\/pre><\/div>\n\n\n
6. Using pandas module<\/h2>\n\n\n\n
In case we are working with the Pandas module, we can use the pandas.drop_duplicates()<\/code> method to remove the duplicates and then convert it into a List, while also preserving the order.<\/p>\n\n\n
\nimport pandas as pd\n\na = [0, 1, 2, 3, 4, 1, 2, 3, 5]\n\npd.Series(a).drop_duplicates().tolist()\n<\/pre><\/div>\n\n\n
Output<\/strong><\/p>\n\n\n
\n[0, 1, 2, 3, 4, 5]\n<\/pre><\/div>\n\n\n
\n\n\n\n
References<\/h2>\n\n\n\n