Re: "In progress" saving of data collected using Reap/Sow
- To: mathgroup at smc.vnet.net
- Subject: [mg66164] Re: "In progress" saving of data collected using Reap/Sow
- From: "Ray Koopman" <koopman at sfu.ca>
- Date: Tue, 2 May 2006 02:43:21 -0400 (EDT)
- References: <e2v5pu$n64$1@smc.vnet.net>
- Sender: owner-wri-mathgroup at wolfram.com
giacomo.ciani at gmail.com wrote:
> Hello to everyone,
>
> I'm quite new to Mathematica, so please be patient if I ask something
> trivial.
> I wrote a little simulation program that need to iterate a calculation
> thousand of times, storing the result of each iteration. At first, I
> used the "Append" function to store data in a list at every iteration,
> but I noticed that this operation become slower and slower as the list
> increases in size, so that appending a single data at the end of a
> quite big list takes a lot of time.
> So I migrated to the reap/sow functions that are written exactly for
> this purpose, ad gives much better performances. My code now is
> something like:
>
> results = Reap[Do[a lot of iterations with almost a call to Sow in
> each]]];
>
> The problem now is that, since the collected data are returned by the
> Reap function only whern it terminates, I have no way to access that
> data if the calculation is in progress, ie if the Do loop is not
> completed. This way, I can't check the status of the calculation nor
> provide backup save of data during the calculation itself, that can
> last for days...
> As you can imagine, using the Append function both this (and others)
> tasks where easily accomplishable, since in every moment I had a list
> whith all the results obtained so far...
>
> Any idea on how I can solve the problem?
>
> Thanks a lot
>
> Giacomo
In[1]:= n = 1*^4
Timing[y = {}; Do[AppendTo[y,i],{i,n}];]
Timing[z = {}; Do[z = {z,i},{i,n}];]
i = Random[Integer,{1,n}]
y[[i]]
Last@Nest[First,z,n-i]
Out[1]= 10000
Out[2]= {1.86 Second,Null}
Out[3]= {0.02 Second,Null}
Out[4]= 9353
Out[5]= 9353
Out[6]= 9353