Re: AppendTo VERY slow
- To: mathgroup at smc.vnet.net
- Subject: [mg35356] Re: AppendTo VERY slow
- From: Tomas Garza <tgarza01 at prodigy.net.mx>
- Date: Tue, 9 Jul 2002 06:49:46 -0400 (EDT)
- References: <000001c22696$c838f100$0200a8c0@HolyCow>
- Sender: owner-wri-mathgroup at wolfram.com
Hi,
You're quite right. I made a foolish mistake in writing down the function f.
In fact it should read
In[1]:=
t = Transpose[{xlist, ylist}];
f[x_, k_] := {x, t[[k]]}
The first argument of f was not used in my earlier message. Then
In[2]:=
garza = Partition[Flatten[Fold[f, {}, Range[n]]], 2];
does the job. It takes roughly twice as much time as your "folder", and I
quite agree that it is just a Do in disguise. Still, it must be realized
that both your approach and mine, as well as most others contributed to this
thread, are equivalent to the matrix Transpose[xlist, ylist] itself. I.e.,
just appending one element after the other gives the original data (your
last message confirms this).
I think that the original question is simply solved by recommending the use
of x = {x, elem} instead of AppendTo. I guess the problem was not meant to
simply construct the original data again, but to append the results of new
calculations to a list (i.e., the {xlist, ylist} was not available from the
beginning, but rather was being constructed step by step, and each new
result was to be appended to the list). Pure speculation on my part, anyway.
Tomas
----- Original Message -----
From: "DrBob" <majort at cox-internet.com>
To: mathgroup at smc.vnet.net
Subject: [mg35356] RE: AppendTo VERY slow
> That use of Fold doesn't actually USE the capabilities of Fold. The
> function f ignores its first argument and you're not using the matrix of
> interest as its third argument either, so you're just using Fold as a Do
> loop. Here are timings for the AppendTo method, yours, and a true use
> of Fold:
>
> n = 10000;
> xlist = Table[Random[Integer, {0, 9}], {n}];
> ylist = Table[Random[Integer, {0, 9}], {n}];
> app := (outlist = {};
> Do[elem = {xlist[[i]], ylist[[i]]};
> AppendTo[outlist, elem], {i, 1, n}])
> Timing[app;]
> folder := Partition[Flatten[
> Fold[{#1, #2} &, {}, Transpose[{xlist, ylist}]]], 2]
> Timing[folder;]
> f[x_, k_] := garza = {garza, {xlist[[k]], ylist[[k]]}}
> garza = {}; garza = Partition[Flatten[Fold[f, {}, Range[n]]], 2]; //
> Timing
> outlist == folder == garza
>
> {9.219 Second,Null}
>
> {0.047 Second,Null}
>
> {0.094 Second,Null}
>
> True
>
> Fold is about twice as fast as your Do loop. Here's a larger test:
>
> n = 40000;
> xlist = Table[Random[Integer, {0, 9}], {n}];
> ylist = Table[Random[Integer, {0, 9}], {n}];
> Timing[folder;]
> Timing[garza = {}; garza = Partition[Flatten[Fold[f, {}, Range[n]]],
> 2];]
> folder == garza
>
> {0.172 Second,Null}
>
> {0.438 Second,Null}
>
> True
>
> Bobby Treat
>
> -----Original Message-----
> From: tgarza01 at prodigy.net.mx [mailto:tgarza01 at prodigy.net.mx]
To: mathgroup at smc.vnet.net
> Sent: Monday, July 08, 2002 2:16 AM
> Subject: [mg35356] AppendTo VERY slow
>
> Indeed, AppendTo is an awfully slow function. It takes exponentially
> longer
> to execute with the list size. I would suggest a simple outlist =
> {outlist,
> elem} with a Flatten and Partition at the end. For example,
>
> In[1]:=
> xlist=Table[Random[Integer,{0,9}],{10000}];
> ylist=Table[Random[Integer,{0,9}],{10000}];
>
> I define a function to do things with Append To:
>
> In[2]:=
> app[n_]:=(
> outlist={};Do[elem={xlist[[i]],ylist[[i]]};
> AppendTo[outlist,elem],{i,1,n}
> ])
>
> In[3]:=
> app[10000]//Timing
>
> Out[3]=
> {45.37 Second,Null}
>
> It takes 45 seconds to process 10,000 elements. Now, with the approach I
> suggest, I define the function f (in order to use Fold, a very powerful
> function):
>
> In[4]:=
> f[x_,n_]:=outlist={outlist,{xlist[[n]],ylist[[n]]}}
>
> In[5]:=
> outlist={};Partition[Flatten[Fold[f,{},Range[10000]]],2];//Timing
>
> Out[5]=
> {0.49 Second,Null}
>
> which produces the same result almost 100 times faster.
>
> Tomas Garza
> Mexico City
>
>
> Original Message:
> -----------------
> From: M.P.Croucher at Sheffield.ac.uk (Mike)
To: mathgroup at smc.vnet.net
> Subject: [mg35356] AppendTo VERY slow
>
>
> I use lists a lot in mathematica and tend to use AppendTo[] a lot in
> my programs. Recently I wrote a function that i call over and over
> again and found that the results were coming very slowly and i MEAN
> slowly. I was doing Fourier Transforms and all kinds of stuff so I
> put it down to those at first but I have just put in a load of Print
> statements just after each part of the function to see what was taking
> so long.
>
> I was amazed to see that the Fourier Transforms were so quick and what
> was actually taking the time was a part of my function that collected
> the results togther in the form I wanted and outputted the result. It
> looks like this
>
> Do[
> elem = {xlist[[count]], ylist[[count]]]};
> AppendTo[outlist, elem];
> , {count, 1, number}
> ];
>
> It seems that as the list grows it gets slower and slower. Any tips
> on a way around this would be greatly appreciated (would speed my life
> up no end)
>
>
> Thank
>
> Mike
>
>
> --------------------------------------------------------------------
> mail2web - Check your email from the web at
> http://mail2web.com/ .
>
>
>
>
>