[Date Index]
[Thread Index]
[Author Index]
Re: Bizarre Running Times
*To*: mathgroup at smc.vnet.net
*Subject*: [mg89335] Re: Bizarre Running Times
*From*: David Bailey <dave at Remove_Thisdbailey.co.uk>
*Date*: Fri, 6 Jun 2008 06:46:29 -0400 (EDT)
*References*: <g27qv8$d2f$1@smc.vnet.net>
Erin Noel wrote:
> Hey,
>
> I have a program that imports data from Excel and then uses that data
> throughout. When the Excel file has only 10 rows, the total running time is
> 90 seconds. As I duplicate these rows in the Excel input file the running
> time (expectedly) increases. At 15 rows, running time is 106 seconds,
> increasing steadily up to 432 seconds with 99 rows. However, when i add a
> 100th row, overall running time oddly drops down to 135 seconds, then
> continues increasing from there with the addition of more rows in a similar
> fashion to the timing increase when the number of rows was less than 100.
> This concerns me because I wonder if my program isn't running some of its
> loops correctly or something. But then at the same time it seems a big
> coincidence that this drop would occur right at 100 rows. Basically, I am
> utterly confused and would very much appreciate any insight anyone may have
> into what's going on. Thank you so much in advance.
>
> Erin
There can certainly be some odd timing effects in Mathematica because
different strategies are used in different situations. For example, if a
certain expression is generated more than once in a calculation,
Mathematica can internally choose (I think) whether to hold one copy or
two. This choice will speed up certain situations and slow down others.
Above a certain size, Mathematica has to choose the option of storing
two copies because the cost of checking that they are the same becomes
excessive.
One possibility is that arrays of homogeneous data (all integers, or all
reals) can be stored in two different forms - packed and unpacked. From
experiment, I notice that Import applied to a csv file of real numbers
always returns an unpacked array. (Use the function
Developer`PackedArrayQ) to test for this). Mathematica may 'notice' this
problem and invisibly convert to the packed format (which is usually
more efficient) at some point. You might want to try reading your data
in thus:
data=Developer`ToPackedArray[Import["..."]];
or even, if the data is to be treated as real, but might contain some
integers:
data=Developer`ToPackedArray[N[Import["..."]]];
Developer`PackedArrayQ[data] will then always return true unless there
is something about the data which prevents packing - such as rows of
unequal length.
BTW, don't worry about using these Developer functions - they have been
available for many years and are used in many places in existing code!
Does your timing include the cost of the Import operation? If it doesn't
then it should be possible to generate the matrix with a bit of
Mathematica code, and post the whole lot here so that people can experiment.
David Bailey
http://www.dbaileyconsultancy.co.uk
Prev by Date:
**Re: Re: Visualization of a list of 3D points coordinates with a perspective**
Next by Date:
**Re: Exporting the output of GraphPlot**
Previous by thread:
**Bizarre Running Times**
Next by thread:
**Import Images and Displaying**
| |