Re: "In progress" saving of data collected using Reap/Sow

*To*: mathgroup at smc.vnet.net*Subject*: [mg66179] Re: "In progress" saving of data collected using Reap/Sow*From*: "Ray Koopman" <koopman at sfu.ca>*Date*: Wed, 3 May 2006 02:44:52 -0400 (EDT)*References*: <e2v5pu$n64$1@smc.vnet.net><e36vft$n7c$1@smc.vnet.net>*Sender*: owner-wri-mathgroup at wolfram.com

Expanding on my previous post, here are three ways to unravel z into a simple list of all the past results. The first way is fastest, but doesn't work if each result itself is a list. In[1]:= n = 8 y = {}; Do[AppendTo[y,result[i]],{i,n}]; y z = {}; Do[z = {z,result[i]},{i,n}]; z Flatten[z] === y Nest[FlattenAt[#,1]&,z,n] === y Table[Last@Nest[First,z,n-i],{i,n}] === y Out[1]= 8 Out[2]= {result[1],result[2],result[3],result[4], result[5],result[6],result[7],result[8]} Out[3]= {{{{{{{{{},result[1]},result[2]},result[3]},result[4]}, result[5]},result[6]},result[7]},result[8]} Out[4]= True Out[5]= True Out[6]= True In[7]:= n = 8 y = {}; Do[AppendTo[y,{i,-i}],{i,n}]; y z = {}; Do[z = {z,{i,-i}},{i,n}]; z Flatten[z] === y Nest[FlattenAt[#,1]&,z,n] === y Table[Last@Nest[First,z,n-i],{i,n}] === y Out[7]= 8 Out[8]= {{1,-1},{2,-2},{3,-3},{4,-4}, {5,-5},{6,-6},{7,-7},{8,-8}} Out[9]= {{{{{{{{{},{1,-1}},{2,-2}},{3,-3}},{4,-4}}, {5,-5}},{6,-6}},{7,-7}},{8,-8}} Out[10]= False Out[11]= True Out[12]= True Ray Koopman wrote: > giacomo.ciani at gmail.com wrote: > > Hello to everyone, > > > > I'm quite new to Mathematica, so please be patient if I ask something > > trivial. > > I wrote a little simulation program that need to iterate a calculation > > thousand of times, storing the result of each iteration. At first, I > > used the "Append" function to store data in a list at every iteration, > > but I noticed that this operation become slower and slower as the list > > increases in size, so that appending a single data at the end of a > > quite big list takes a lot of time. > > So I migrated to the reap/sow functions that are written exactly for > > this purpose, ad gives much better performances. My code now is > > something like: > > > > results = Reap[Do[a lot of iterations with almost a call to Sow in > > each]]]; > > > > The problem now is that, since the collected data are returned by the > > Reap function only whern it terminates, I have no way to access that > > data if the calculation is in progress, ie if the Do loop is not > > completed. This way, I can't check the status of the calculation nor > > provide backup save of data during the calculation itself, that can > > last for days... > > As you can imagine, using the Append function both this (and others) > > tasks where easily accomplishable, since in every moment I had a list > > whith all the results obtained so far... > > > > Any idea on how I can solve the problem? > > > > Thanks a lot > > > > Giacomo > > > In[1]:= n = 1*^4 > Timing[y = {}; Do[AppendTo[y,i],{i,n}];] > Timing[z = {}; Do[z = {z,i},{i,n}];] > i = Random[Integer,{1,n}] > y[[i]] > Last@Nest[First,z,n-i] > > Out[1]= 10000 > Out[2]= {1.86 Second,Null} > Out[3]= {0.02 Second,Null} > Out[4]= 9353 > Out[5]= 9353 > Out[6]= 9353