memory use
- To: mathgroup at smc.vnet.net
- Subject: [mg4902] memory use
- From: rustybel at foothill.net
- Date: Fri, 4 Oct 1996 00:17:39 -0400
- Sender: owner-wri-mathgroup at wolfram.com
Having posted a question about excessive kernel memory use when writing a
relatively small file to disk, I received very much helpful information
from various people. Thanks to Mark Evans, Harald Berndt, and Paul Rubin
for their especially detailed remarks.
The most immediately useful method offered, which works admirably well, was
supplied by Rolf Mertig. His remarks and code are as follows:
"Mathematica has a well-known efficiency problem in reading in and
writing out larger files.
"Of course everybody finding this bug should report to support at wri.com,
so that it might get fixed still in some new version.
"In the meantime probably the thing to do is use a simple procedural
Do-loop, in good old Fortran-style:
BeginPackage["PutLargeMatrix`"];
PutLargeMatrix::usage=
"PutLargeMatrix[{{x1, x2, ...}, {...}, {y1,y2, ...}}, filename]
writes the matrix {{...}} to a file.
The memory-consumption is much less.";
Begin["`Public`"];
PutLargeMatrix[{x_List, y__List, z_List}, fil_String] := Block[{},
OpenWrite[fil]; WriteString[fil,"{\n"]; PutAppend[x, fil];
Do[WriteString[fil,","];
PutAppend[{y}[[i]], fil];
(* comment this line out, evtl.*) WriteString["stdout",i," "];
, {i,Length[{y}]}
];
WriteString[fil,","]; PutAppend[z, fil]; WriteString[fil,"}\n"];
Close[fil] ];
End[]; EndPackage[];
(* on a HP720, running Mma 2.2, this examples takes about 5.5 minutes and
returns 3843712 for MaxMemoryUsed[]
(* this puts out a 491 x 1091 matrix *)
PutLargeMatrix[Table[Mod[x,y],{y,491},{x,1091}], "longtest"]
*)."
Rolf's method consumes little memory at all in the course of actually
writing the file. It does seem to me that some such method should be
incorportated as a standard Mathematica function in future releases.
Russell Towle
Giant Gap Press
P.O. Box 141
Dutch Flat, California 95714
------------------------------
Voice: (916) 389-2872
e-mail: rustybel at foothill.net
------------------------------
==== [MESSAGE SEPARATOR] ====