Services & ResourcesWolfram ForumsMathGroup Archive

[Date Index] [Thread Index] [Author Index]

Re: Eliminate duplicates using Union [] in a

• To: mathgroup at smc.vnet.net
• Subject: [mg48070] Re: [mg48063] Eliminate duplicates using Union [] in a
• From: Andrzej Kozlowski <akoz at mimuw.edu.pl>
• Date: Tue, 11 May 2004 05:19:53 -0400 (EDT)
• References: <200405101051.GAA13867@smc.vnet.net>
• Sender: owner-wri-mathgroup at wolfram.com

On 10 May 2004, at 19:51, Mark Coleman wrote:

> I know that if one applies the Union command to a list of values, it
> will return the list with duplicates removed. Can I use the Union
> command to eliminate duplicate when I have a list of data "records"?
> Specifically, I have a large data set that I read into a list on a
> row-oriented basis. Each element of the list consists of another list
> of about 30 different fields. Two of these fields are geographic
> indicators. If two or more records have identical values in these two
> fields, then the full records are considered duplicates and need to be
> identified and removed from the larger list (or at least marked for
> removal). Can Union do this?
>
> Thanks,
>
> -Mark
>
>
>
One way use the option SameTest in Union, which lets you specify the 
crieterion for "sameness" of objects in your list. For example 
consider:


myList = {{a, b, c}, {a, b, d}, {e, f, g}};

Suppose we want to regard two elements of myList as the same if they 
have the same first two elements. You can then "remove duplicates" as 
follows:


Union[myList, SameTest ->
    (Take[#1, 2] === Take[#2, 2] & )]


{{a, b, c}, {e, f, g}

This is method is very flexible and will work well essenitally with any 
symmetric and transitive notion of sameness (if I recall correclty this 
was not quite the case with early versions of Mathematica).

Andrzej Kozlowski
Chiba, Japan
http://www.mimuw.edu.pl/~akoz/

• References:
  • Eliminate duplicates using Union [] in a
    • From: Mark Coleman <mark@markscoleman.com>