Re: Speeding UP Indexing and Joining ofDifferentSizeRectangular Matrixes
- To: mathgroup at smc.vnet.net
- Subject: [mg52453] Re: [mg52414] Speeding UP Indexing and Joining ofDifferentSizeRectangular Matrixes
- From: Tomas Garza <tgarza01 at prodigy.net.mx>
- Date: Sat, 27 Nov 2004 01:41:35 -0500 (EST)
- Sender: owner-wri-mathgroup at wolfram.com
Perhaps something could be done. Please explain your problem in more detail.
Never mind your program (what is loc01?). What do you mean by joining rows?
Give a small example with small matrices (say 5 x 2 or something like that).
What are your present run times?
Tomas Garza
Mexico City
----- Original Message -----
From: "Benedetto Bongiorno" <bbongiorno at attglobal.net>
To: mathgroup at smc.vnet.net
Subject: [mg52453] [mg52414] Speeding UP Indexing and Joining of
DifferentSizeRectangular Matrixes
>
> Fellow MathGroup,
>
> I have been using Mathematica for financial analysis purposes and have
> been
> developing note book programs for about 5 years.
> My skills at this are self taught with help from Wolfram training and
> support.
> The largest challenge has been the speed in the analysis of large data
> sets.
> The following is an example of a routine that takes many hours.
> PLEASE HELP AND SHOW HOW I CAN IMPROVE THE ROUTINE TO MAKE THE RUN TIME
> SHORTER.
>
> Equipment HP XP 3.24 processor 2 Gigs
> Mathematica 5.01
> Data set a= 257470 by 40, Mixed numeric and string fields, but each field
> (column) is either or numeric or string
> Data set b= 258705 by 5, All fields are numeric
>
> Objective: RowJoin the rows from each data set that have the same ID
> field
> in their corresponding column one.
>
> Thank you and Happy Holidays
>
> ROUTINE
> Create Index By Invoice ID
>
> firstCol=loc01[[1]];
>
> lastCol =loc01[[1]];
>
> aa = Transpose[Take[Transpose[a],{firstCol, lastCol}]];
>
> Length[aa]
>
> 257470
>
> firstCol=loc04[[1]];
>
> lastCol =loc04[[1]];
>
> bb = Transpose[Take[Transpose[b],{firstCol, lastCol}]];
>
> Length[bb]
>
> 258705
>
> idx=Intersection[aa,bb];
>
> Length[idx]
>
> 257249
>
> n=Length[idx]+1
>
> 257250
>
> Locate Position Of Each Record In aTable
>
> ans01={};
>
> For[i=1,i<n,i++,
>
> step1 = Position[aa,idx[[i]]];
>
> AppendTo[ans01,step1]]
>
> ans01=Flatten[ans01,1];
>
> Locate Position Of Each Record In bTable
>
> ans02={};
>
> For[i=1,i<n,i++,
>
> step1 = Position[bb,idx[[i]]];
>
> AppendTo[ans02,step1]]
>
> ans02=Flatten[ans02,1];
>
> Extract a Records by Index
>
> ans01 =Extract[currentBalance,ans01];
>
> Dimensions[ans01]
>
> Flatten If Not A Matrix
>
> If[MatrixQ[ans01],ans01=ans01,ans01=Flatten[ans01,1]];
>
> Dimensions[ans01]
>
> Extract b Records by Index
>
> ans02 =Extract[interestBalance,ans02];
>
> Dimensions[ans02]
>
> Flatten If Not A Matrix
>
> If[MatrixQ[ans02],ans02=ans02,ans02=Flatten[ans02,1]];
>
> Dimensions[ans02]
>
> ans01=matsort[ans01,loc01[[1]]];
>
> ans02=matsort[ans02,loc04[[1]]];
>
> noteTerms=RowJoin[ans02,ans01];
>
> Dimensions[noteTerms]
>
>
>