[Date Index] [Thread Index] [Author Index]

Re: Bizarre Running Times

• To: mathgroup at smc.vnet.net
• Subject: [mg89335] Re: Bizarre Running Times
• From: David Bailey <dave at Remove_Thisdbailey.co.uk>
• Date: Fri, 6 Jun 2008 06:46:29 -0400 (EDT)
• References: <g27qv8$d2f$1@smc.vnet.net>

Erin Noel wrote:
> Hey,
> 
> I have a program that imports data from Excel and then uses that data
> throughout. When the Excel file has only 10 rows, the total running time is
> 90 seconds. As I duplicate these rows in the Excel input file the running
> time (expectedly) increases. At 15 rows, running time is 106 seconds,
> increasing steadily up to 432 seconds with 99 rows. However, when i add a
> 100th row, overall running time oddly drops down to 135 seconds, then
> continues increasing from there with the addition of more rows in a similar
> fashion to the timing increase when the number of rows was less than 100.
> This concerns me because I wonder if my program isn't running some of its
> loops correctly or something. But then at the same time it seems a big
> coincidence that this drop would occur right at 100 rows. Basically, I am
> utterly confused and would very much appreciate any insight anyone may have
> into what's going on. Thank you so much in advance.
> 
> Erin

There can certainly be some odd timing effects in Mathematica because 
different strategies are used in different situations. For example, if a 
certain expression is generated more than once in a calculation, 
Mathematica can internally choose (I think) whether to hold one copy or 
two. This choice will speed up certain situations and slow down others. 
Above a certain size, Mathematica has to choose the option of storing 
two copies because the cost of checking that they are the same becomes 
excessive.

One possibility is that arrays of homogeneous data (all integers, or all 
reals) can be stored in two different forms - packed and unpacked. From 
experiment, I notice that Import applied to a csv file of real numbers 
always returns an unpacked array. (Use the function 
Developer`PackedArrayQ) to test for this). Mathematica may 'notice' this 
problem and invisibly convert to the packed format (which is usually 
more efficient) at some point. You might want to try reading your data 
in thus:

data=Developer`ToPackedArray[Import["..."]];

or even, if the data is to be treated as real, but might contain some 
integers:

data=Developer`ToPackedArray[N[Import["..."]]];

Developer`PackedArrayQ[data] will then always return true unless there 
is something about the data which prevents packing - such as rows of 
unequal length.

BTW, don't worry about using these Developer functions - they have been 
available for many years and are used in many places in existing code!

Does your timing include the cost of the Import operation? If it doesn't 
then it should be possible to generate the matrix with a bit of 
Mathematica code, and post the whole lot here so that people can experiment.

David Bailey
http://www.dbaileyconsultancy.co.uk