DeleteDuplicates uses a lot of memory
- To: mathgroup at smc.vnet.net
- Subject: [mg94644] DeleteDuplicates uses a lot of memory
- From: Nacho <ncc1701zzz at gmail.com>
- Date: Thu, 18 Dec 2008 07:23:53 -0500 (EST)
Hello.
I've noticed that DeleteDuplicates[list] uses a lot of memory if most
of the elements in the list are different.
>From the example in Help:
list = RandomInteger[10^5, 10^7];
{First[Timing[DeleteDuplicates[list]]], First[Timing[Union[list]]]}
This works fine (but MaxMemoryUsed[] is increased in 120Mb!), but if
you change the RandomInteger margin:
list = RandomInteger[10^7, 10^7];
{First[Timing[DeleteDuplicates[list]]], First[Timing[Union[list]]]}
Takes a lot of memory. I have unable to complete the command in my
modest computer. Union takes some time but it's OK.
I suppose that DeleteDuplicates is using some kind of table to see if
every element in the list is new or not, but it really fills up the
memory!
Any idea? Is this a bug?
Thanks.