Services & ResourcesWolfram ForumsMathGroup Archive

[Date Index] [Thread Index] [Author Index]

Speeding UP Indexing and Joining of Different Size Rectangular Matrixes

• To: mathgroup at smc.vnet.net
• Subject: [mg52414] Speeding UP Indexing and Joining of Different Size Rectangular Matrixes
• From: "Benedetto Bongiorno" <bbongiorno at attglobal.net>
• Date: Fri, 26 Nov 2004 01:04:53 -0500 (EST)
• Sender: owner-wri-mathgroup at wolfram.com

Fellow MathGroup,

I have been using Mathematica for financial analysis purposes and have been
developing note book programs for about 5 years.
My skills at this are self taught with help from Wolfram training and support.
The largest challenge has been the speed in the analysis of large data sets.
The following is an example of a routine that takes many hours.
PLEASE HELP AND SHOW HOW I CAN IMPROVE THE ROUTINE TO MAKE THE RUN TIME
SHORTER.

Equipment HP XP 3.24 processor 2 Gigs
Mathematica 5.01
Data set a= 257470 by 40, Mixed numeric and string fields, but each field
(column) is either or numeric or string
Data set b= 258705 by 5, All fields are numeric

Objective:  RowJoin the rows from each data set that have the same ID field
in their corresponding column one.

Thank you and Happy Holidays

ROUTINE
Create Index By Invoice ID

firstCol=loc01[[1]];

lastCol =loc01[[1]];

aa = Transpose[Take[Transpose[a],{firstCol, lastCol}]];

Length[aa]

257470

firstCol=loc04[[1]];

lastCol =loc04[[1]];

bb = Transpose[Take[Transpose[b],{firstCol, lastCol}]];

Length[bb]

258705

idx=Intersection[aa,bb];

Length[idx]

257249

n=Length[idx]+1

257250

Locate Position Of Each Record In aTable

ans01={};

For[i=1,i<n,i++,

step1 = Position[aa,idx[[i]]];

AppendTo[ans01,step1]]

ans01=Flatten[ans01,1];

Locate Position Of Each Record In bTable

ans02={};

For[i=1,i<n,i++,

step1 = Position[bb,idx[[i]]];

AppendTo[ans02,step1]]

ans02=Flatten[ans02,1];

Extract a Records by Index

ans01 =Extract[currentBalance,ans01];

Dimensions[ans01]

Flatten If Not A Matrix

If[MatrixQ[ans01],ans01=ans01,ans01=Flatten[ans01,1]];

Dimensions[ans01]

Extract b Records by Index

ans02 =Extract[interestBalance,ans02];

Dimensions[ans02]

Flatten If Not A Matrix

If[MatrixQ[ans02],ans02=ans02,ans02=Flatten[ans02,1]];

Dimensions[ans02]

ans01=matsort[ans01,loc01[[1]]];

ans02=matsort[ans02,loc04[[1]]];

noteTerms=RowJoin[ans02,ans01];

Dimensions[noteTerms]

• Follow-Ups:
  • Re: Speeding UP Indexing and Joining of Different Size Rectangular Matrixes
    • From: DrBob <drbob@bigfoot.com>