Re: Bizarre Running Times
- To: mathgroup at smc.vnet.net
- Subject: [mg89335] Re: Bizarre Running Times
- From: David Bailey <dave at Remove_Thisdbailey.co.uk>
- Date: Fri, 6 Jun 2008 06:46:29 -0400 (EDT)
- References: <g27qv8$d2f$1@smc.vnet.net>
Erin Noel wrote: > Hey, > > I have a program that imports data from Excel and then uses that data > throughout. When the Excel file has only 10 rows, the total running time is > 90 seconds. As I duplicate these rows in the Excel input file the running > time (expectedly) increases. At 15 rows, running time is 106 seconds, > increasing steadily up to 432 seconds with 99 rows. However, when i add a > 100th row, overall running time oddly drops down to 135 seconds, then > continues increasing from there with the addition of more rows in a similar > fashion to the timing increase when the number of rows was less than 100. > This concerns me because I wonder if my program isn't running some of its > loops correctly or something. But then at the same time it seems a big > coincidence that this drop would occur right at 100 rows. Basically, I am > utterly confused and would very much appreciate any insight anyone may have > into what's going on. Thank you so much in advance. > > Erin There can certainly be some odd timing effects in Mathematica because different strategies are used in different situations. For example, if a certain expression is generated more than once in a calculation, Mathematica can internally choose (I think) whether to hold one copy or two. This choice will speed up certain situations and slow down others. Above a certain size, Mathematica has to choose the option of storing two copies because the cost of checking that they are the same becomes excessive. One possibility is that arrays of homogeneous data (all integers, or all reals) can be stored in two different forms - packed and unpacked. From experiment, I notice that Import applied to a csv file of real numbers always returns an unpacked array. (Use the function Developer`PackedArrayQ) to test for this). Mathematica may 'notice' this problem and invisibly convert to the packed format (which is usually more efficient) at some point. You might want to try reading your data in thus: data=Developer`ToPackedArray[Import["..."]]; or even, if the data is to be treated as real, but might contain some integers: data=Developer`ToPackedArray[N[Import["..."]]]; Developer`PackedArrayQ[data] will then always return true unless there is something about the data which prevents packing - such as rows of unequal length. BTW, don't worry about using these Developer functions - they have been available for many years and are used in many places in existing code! Does your timing include the cost of the Import operation? If it doesn't then it should be possible to generate the matrix with a bit of Mathematica code, and post the whole lot here so that people can experiment. David Bailey http://www.dbaileyconsultancy.co.uk