Re: Need to Speed up Position[]
- To: mathgroup at smc.vnet.net
- Subject: [mg110706] Re: Need to Speed up Position[]
- From: Leonid Shifrin <lshifr at gmail.com>
- Date: Sat, 3 Jul 2010 08:15:52 -0400 (EDT)
Hi,
This is a follow-up to my previous post.
I did not quite get your original goal before and my previous post was
somewhat messy. Here is the code which will probably do what you need:
The test list
myList = RandomInteger[{1, 5555}, #] & /@ {{19808, 5}, {7952,
5}, {7952, 5}};
The code:
Clear[mergeSameDates];
mergeSameDates[lst_List] :=
With[{pos = PositionsOfSame @@ lst[[All, All, 5]]},
With[{trpos = Transpose[pos]},
Flatten[
Transpose@
Table[lst[[i, #]] & /@ trpos[[i]], {i, 1,
Length[lst]}], {{1}, {2, 3}}]]];
It uses the function PositionsOfSame, described in my previous post. You use
it as follows:
In[4]:= (res = mergeSameDates[myList]); // Timing
Out[4]= {0.831, Null}
Regards,
Leonid
On Thu, Jul 1, 2010 at 5:28 AM, Garapata <warsaw95826 at mypacks.net> wrote:
> I have a large nested list, "myList"
>
> It has 3 sublists with the following dimensions:
>
> Dimensions/@ myList
>
> {{19808, 5}, {7952, 5}, {7952, 5}}
>
> The 5th position (i.e., column) in each of the sublists has
> SQLDateTime[]s
> (This may or may not affect what I need, but I thought everyone should
> know).
>
> myIntersection = Intersection @@ (myList[[All, All, 5]]);
>
> gives me the SQLDateTimes[]s common to all sublists. I get 3954
> common elements.
>
> Length[myIntersection]
>
> 3954
>
> All of the above works great and runs very fast.
>
> I then find the positions in myList where all the common
> SQLDateTimes[]s occur and then use Extract pull them out into a new
> list
>
> myPositions = Drop[(Position[data, #] & /@ myIntersection), None,
> None, -1];
>
> myOutput = Extract[myList, #] & /@ myPositions;
>
> I end up with just what I need, which in this case gives me 3954 rows
> of {9, 5} sublists. This occurs because myList[[1]] has 5 occurrences
> of each common date element and sublists myList[[2]] and myList[[3]]
> each have 2 occurrences of each common date element.
>
> The Extract[] runs very fast.
>
> My problem =85. the Position[] runs very very slow (over 90 seconds on a
> dual core iMac).
>
> All the code together:
>
> myIntersection = Intersection @@ (myList[[All, All, 5]]);
> myPositions = Drop[(Position[data, #] & /@ myIntersection), None,
> None, -1];
> myOutput = Extract[myList, #] & /@ myPositions;
>
> So, does anyone know a way to speed up:
>
> myPositions = Drop[(Position[data, #] & /@ myIntersection), None,
> None, -1]; ?
>
> Or can anyone suggest another approach to doing this that could run
> faster.
>
> Patterns?
> ParallelMap?
> Parallelize?
> Sorting?
> Changing SQLDateTimes to DateList[]s before calculating myPositions?
>
> Not clear what to try.
> Please advise.
>
> Thanks.
>
>