Re: "In progress" saving of data collected using Reap/Sow
- To: mathgroup at smc.vnet.net
- Subject: [mg66147] Re: "In progress" saving of data collected using Reap/Sow
- From: bghiggins at ucdavis.edu
- Date: Sun, 30 Apr 2006 04:22:28 -0400 (EDT)
- References: <e2v5pu$n64$1@smc.vnet.net>
- Sender: owner-wri-mathgroup at wolfram.com
Giacomo,
I am not familiar with your actual code, but if you could break up your
problem into a set of subproblems that can be restarted then you could
write blocks of your data to file. Here is how one could go about
doing it:
First, open up a file to write to:
strm = OpenWrite["/Users/brian/Desktop/temps.dat"]
Now suppose my problem was to add a random number to the previous
element in a list and do it 300 times. The Reap/Sow strategy saves the
intremediate results in a list
a=0;Reap[Do[a = a + Random[];Sow[a], {i, 1, 300}]]
Let us suppose we want to back-up intermediate results to a file after
every 100 iterations Again we implement a reap/sow strategy but now
break up the problem into 3 sub blocks of 100 iterations nad place the
Reap/Sow stategy inside the ionner loop
a = 0; Do[Reap[Do[a = a + Random[]; Sow[a, strm], {i, 1, 100}], _,
Write], {3}]
The key idea is to used the channel name "strm" as a tag for Sow, and
then reference this tag in Reap, as shown above. In this way after
every 100 interations of the inner Do loop, data is written to my file.
Hope this gives you some ideas- this is all I could think of. Perhaps
others on the mathgroup have a better idea.
Cheers,
Brian
giacomo.ciani at gmail.com wrote:
> Hello to everyone,
>
> I'm quite new to Mathematica, so please be patient if I ask something
> trivial.
> I wrote a little simulation program that need to iterate a calculation
> thousand of times, storing the result of each iteration. At first, I
> used the "Append" function to store data in a list at every iteration,
> but I noticed that this operation become slower and slower as the list
> increases in size, so that appending a single data at the end of a
> quite big list takes a lot of time.
> So I migrated to the reap/sow functions that are written exactly for
> this purpose, ad gives much better performances. My code now is
> something like:
>
> results = Reap[Do[a lot of iterations with almost a call to Sow in
> each]]];
>
> The problem now is that, since the collected data are returned by the
> Reap function only whern it terminates, I have no way to access that
> data if the calculation is in progress, ie if the Do loop is not
> completed. This way, I can't check the status of the calculation nor
> provide backup save of data during the calculation itself, that can
> last for days...
> As you can imagine, using the Append function both this (and others)
> tasks where easily accomplishable, since in every moment I had a list
> whith all the results obtained so far...
>
> Any idea on how I can solve the problem?
>
> Thanks a lot
>
> Giacomo