memory use
- To: mathgroup at smc.vnet.net
- Subject: [mg4902] memory use
- From: rustybel at foothill.net
- Date: Fri, 4 Oct 1996 00:17:39 -0400
- Sender: owner-wri-mathgroup at wolfram.com
Having posted a question about excessive kernel memory use when writing a relatively small file to disk, I received very much helpful information from various people. Thanks to Mark Evans, Harald Berndt, and Paul Rubin for their especially detailed remarks. The most immediately useful method offered, which works admirably well, was supplied by Rolf Mertig. His remarks and code are as follows: "Mathematica has a well-known efficiency problem in reading in and writing out larger files. "Of course everybody finding this bug should report to support at wri.com, so that it might get fixed still in some new version. "In the meantime probably the thing to do is use a simple procedural Do-loop, in good old Fortran-style: BeginPackage["PutLargeMatrix`"]; PutLargeMatrix::usage= "PutLargeMatrix[{{x1, x2, ...}, {...}, {y1,y2, ...}}, filename] writes the matrix {{...}} to a file. The memory-consumption is much less."; Begin["`Public`"]; PutLargeMatrix[{x_List, y__List, z_List}, fil_String] := Block[{}, OpenWrite[fil]; WriteString[fil,"{\n"]; PutAppend[x, fil]; Do[WriteString[fil,","]; PutAppend[{y}[[i]], fil]; (* comment this line out, evtl.*) WriteString["stdout",i," "]; , {i,Length[{y}]} ]; WriteString[fil,","]; PutAppend[z, fil]; WriteString[fil,"}\n"]; Close[fil] ]; End[]; EndPackage[]; (* on a HP720, running Mma 2.2, this examples takes about 5.5 minutes and returns 3843712 for MaxMemoryUsed[] (* this puts out a 491 x 1091 matrix *) PutLargeMatrix[Table[Mod[x,y],{y,491},{x,1091}], "longtest"] *)." Rolf's method consumes little memory at all in the course of actually writing the file. It does seem to me that some such method should be incorportated as a standard Mathematica function in future releases. Russell Towle Giant Gap Press P.O. Box 141 Dutch Flat, California 95714 ------------------------------ Voice: (916) 389-2872 e-mail: rustybel at foothill.net ------------------------------ ==== [MESSAGE SEPARATOR] ====