MathGroup Archive 2004

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Eliminate duplicates using Union [] in a

  • To: mathgroup at
  • Subject: [mg48070] Re: [mg48063] Eliminate duplicates using Union [] in a
  • From: Andrzej Kozlowski <akoz at>
  • Date: Tue, 11 May 2004 05:19:53 -0400 (EDT)
  • References: <>
  • Sender: owner-wri-mathgroup at

On 10 May 2004, at 19:51, Mark Coleman wrote:

> I know that if one applies the Union command to a list of values, it
> will return the list with duplicates removed. Can I use the Union
> command to eliminate duplicate when I have a list of data "records"?
> Specifically, I have a large data set that I read into a list on a
> row-oriented basis. Each element of the list consists of another list
> of about 30 different fields. Two of these fields are geographic
> indicators. If two or more records have identical values in these two
> fields, then the full records are considered duplicates and need to be
> identified and removed from the larger list (or at least marked for
> removal). Can Union do this?
> Thanks,
> -Mark
One way use the option SameTest in Union, which lets you specify the 
crieterion for "sameness" of objects in your list. For example 

myList = {{a, b, c}, {a, b, d}, {e, f, g}};

Suppose we want to regard two elements of myList as the same if they 
have the same first two elements. You can then "remove duplicates" as 

Union[myList, SameTest ->
    (Take[#1, 2] === Take[#2, 2] & )]

{{a, b, c}, {e, f, g}

This is method is very flexible and will work well essenitally with any 
symmetric and transitive notion of sameness (if I recall correclty this 
was not quite the case with early versions of Mathematica).

Andrzej Kozlowski
Chiba, Japan

  • Prev by Date: Re: Simple question
  • Next by Date: Re: how to save a mathematica-made imagebox.image as jpg, gif, bmp in c#/.netlink?
  • Previous by thread: Eliminate duplicates using Union [] in a
  • Next by thread: Re: Eliminate duplicates using Union [] in a