Re: Export
- To: mathgroup at smc.vnet.net
- Subject: [mg110233] Re: Export
- From: Clint <clint.zeringue at kirtland.af.mil>
- Date: Wed, 9 Jun 2010 07:21:51 -0400 (EDT)
Hi Bill,
Thank you for youre response. I just read this post and in the meantime I found this work around:
read a file and skip so many records down.. (here i skip so many iterations...)
strm = OpenRead[ToFileName[{NotebookDirectory[]}, "1laserGrid.txt"]];
If[iter != 1, Skip[strm, Record, (iter - 1)*Nz]];
EL = ReadList[strm, Expression];
Close[strm];
append to a file
strm = OpenAppend[ToFileName[{NotebookDirectory[]}, "1laserGrid.txt"],
FormatType -> OutputForm];
WriteString[strm,
"\n" <> StringDrop[
StringDrop[
StringReplace[ToString[ELNewNew, InputForm], "," -> "\n"],
1], -1]];
Close[strm];
Both your solution and this one seem to give me the same problem which I will describe below:
The reason for using export and import is that I max out the 32 GB RAM on my PC. So to keep RAM down I use file I/O.
While the "write" stream takes no time at all and doesn't depend on the current file size the read command gets bogged down as the file gets larger and larger... I naively thought that using "skip" in the read would prevent the processor from reading in the entire file which starts to take a very long time as the file sizes approch 100,000 KB's...
This is making simulations virtually imposibble to run since they are taking 10 hours primarily because I keep looping through this read command everytime I propogate my PDE ...and each one is taking a minute or so...
I'm at a loss on what to do here.....
I have no choice but to use file I/O do to RAM limitations but i don't see a way around my problem there either :(
One way I thought of was to maintain two data files.. "one where i store all my data" and the second one "where I just store the last 2 iterations of data since that is all I need to propogate my pde forward in time anyway".. However, I thought maybe I would not have to do that if I could use non-sequential data pulling from a file in Mathematica but I guess that isn't possible?
- Follow-Ups:
- Re: Export
- From: "Mitch Stonehocker" <mitch@aitoconsulting.com>
- Re: Export