Services & Resources / Wolfram Forums / MathGroup Archive
-----

MathGroup Archive 2010

[Date Index] [Thread Index] [Author Index]

Search the Archive

Need to Speed up Position[]

  • To: mathgroup at smc.vnet.net
  • Subject: [mg110671] Need to Speed up Position[]
  • From: Garapata <warsaw95826 at mypacks.net>
  • Date: Thu, 1 Jul 2010 08:28:20 -0400 (EDT)

I have a large nested list, "myList"

It has 3 sublists with the following dimensions:

   Dimensions/@ myList

   {{19808, 5}, {7952, 5}, {7952, 5}}

The 5th position (i.e., column) in each of the sublists has
SQLDateTime[]s
(This may or may not affect what I need, but I thought everyone should
know).

   myIntersection = Intersection @@ (myList[[All, All, 5]]);

gives me the SQLDateTimes[]s common to all sublists.  I get 3954
common elements.

   Length[myIntersection]

   3954

All of the above works great and runs very fast.

I then find the positions in myList where all the common
SQLDateTimes[]s occur and then use Extract pull them out into a new
list

	myPositions = Drop[(Position[data, #] & /@ myIntersection), None,
None, -1];

	myOutput = Extract[myList, #] & /@ myPositions;

I end up with just what I need, which in this case gives me 3954 rows
of {9, 5} sublists. This occurs because myList[[1]] has 5 occurrences
of each common date element and sublists myList[[2]] and myList[[3]]
each have 2 occurrences of each common date element.

The Extract[] runs very fast.

My problem =85. the Position[] runs very very slow (over 90 seconds on a
dual core iMac).

All the code together:

   myIntersection = Intersection @@ (myList[[All, All, 5]]);
   myPositions = Drop[(Position[data, #] & /@ myIntersection), None,
None, -1];
   myOutput = Extract[myList, #] & /@ myPositions;

So, does anyone know a way to speed up:

   myPositions = Drop[(Position[data, #] & /@ myIntersection), None,
None, -1]; ?

Or can anyone suggest another approach to doing this that could run
faster.

Patterns?
ParallelMap?
Parallelize?
Sorting?
Changing SQLDateTimes to DateList[]s before calculating myPositions?

Not clear what to try.
Please advise.

Thanks.


  • Prev by Date: Re: Run Package vs Needs
  • Next by Date: Re: A modified StyleSheet results in FontSize fluctuations
  • Previous by thread: Re: Lauching application from an icon
  • Next by thread: Re: Need to Speed up Position[]