MathGroup Archive 2012

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Find Position of many elements in a large list.


Am 14.08.2012 11:05, schrieb benp84 at gmail.com:
> I have a sorted, 1-dimensional list X of 1,000,000 integers, and a sorted, 1-dimensional list Y of 10,000 integers.  Most, but not all, of the elements of Y are also elements of X.  I'd like to know the positions of the elements in X that are also in Y.  What's the fastest way to compute this?
>
> I have an algorithm in mind but it requires lots of custom code and I'm wondering if there's a clever way to do it with built-in functions.  Thanks.
>

You dont need any "code" at all.

Condsider this example executed on an i7 quad core notebook

data = RandomInteger[10000000, 500000];
pattern = RandomInteger[10000000, 10000];

patterninuse = Intersection[data, pattern];

Timing[pointers = Position[data, #] & /@ patterninuse;]
{17.471999999999998`, Null}

Now we Parallize by splitting the target list in two.

Four kernels get launched

{rg1, rg2} = Partition[data, Length[data]/2];
t1 = AbsoluteTime[];
Parallelize[
  pt1 = Position[rg1, #] & /@ patterninuse;
  pt2 = Position[rg2, #] & /@ patterninuse;
  pt = Join[pt1, pt2 + Length[pt1]];
  ]
AbsoluteTime[] - t1

4.7736084`8.130391782549879


Timing[] with Parallelize[] doesnt work.  Splitting a list of 1000000 in 
four partitions (plus a little error r3 instead rg3) sends the machine 
into nowhere.

Since memory reaches the maximum of 8 Gb and after some minutes even 
mouse interrupt is freezing and screen becomes black there may be a 
parallel problem using the graphic processor.

-- 

Roland Franzius




  • Prev by Date: Problem with Fit command
  • Next by Date: Re: Removing Outliers from List
  • Previous by thread: Re: Find Position of many elements in a large list.
  • Next by thread: Re: Find Position of many elements in a large list.