Re: Huge memory leaks when importing two-dimensional array in Mathematica
- To: mathgroup at smc.vnet.net
- Subject: [mg103457] Re: Huge memory leaks when importing two-dimensional array in Mathematica
- From: "Alexey Popkov" <lehin.p at gmail.com>
- Date: Tue, 22 Sep 2009 07:10:04 -0400 (EDT)
- References: <h9927o$1nm$1@smc.vnet.net> <13796d5e-6a12-40ab-acf8-ca0b4a007b09@z34g2000vbl.googlegroups.com>
- Reply-to: "Alexey Popkov" <lehin.p at gmail.com>
Hello,
I have tried to use WDX file format but it gives another (even more ban)
problem with memory. In this case the problem may be seen only by the
function MaxMemoryUsed[]. Consider the following:
In[1]:= MemoryInUse[]
t = RandomReal[{-1, 1}, {1000000, 2}];
MemoryInUse[]
SetDirectory["C:/"]
Export["temp.wdx", t]
MemoryInUse[]
MaxMemoryUsed[]
Quit[]
Out[1]= 10040648
Out[3]= 26058304
Out[4]= "C:\\"
Out[5]= "temp.wdx"
Out[6]= 27813840
Out[7]= 216689416
Maximum memory used for exporting 16Mb matrix is more than 206Mb! It means
that I even could not export my matrix having size more than 512Mb!
The only advantage of using WDX file format is that Import requires less
memory than Get:
In[1]:= MemoryInUse[]
SetDirectory["C:/"]
t = Import["temp.wdx"];
MemoryInUse[]
MaxMemoryUsed[]
Out[1]= 10039304
Out[2]= "C:\\"
Out[4]= 27902936
Out[5]= 116955560
In[8]:= MemoryInUse[]
SetDirectory["C:/"]
t = << temp.dat;
MemoryInUse[]
MaxMemoryUsed[]
Quit[]
Out[8]= 26637808
Out[9]= "C:\\"
Out[11]= 102639768
Out[12]= 151706272
Is there a way to force Get to return a packed array? Or is it possible to
write may own Get function that will work as Get in Mathematica 5.2?
----- Original Message -----
From: "Vince" <blueschi at gmail.com>
To: "Alexey" <lehin.p at gmail.com>
Sent: Tuesday, September 22, 2009 10:45 AM
Subject: [mg103457] Re: Huge memory leaks when importing two-dimensional array in
Mathematica
On Sep 21, 7:28 pm, Alexey <lehi... at gmail.com> wrote:
> Hello,
> Consider the following (evaluated in Mathematica 7.01):
>
> In[1]:= MemoryInUse[]
> t = RandomReal[{-1, 1}, {1000000, 2}];
> MemoryInUse[]
> SetDirectory["C:/"]
> t >> temp.dat
> MemoryInUse[]
> Quit[]
> Out[1]= 10040344
> Out[3]= 26058000
> Out[4]= "C:\\"
> Out[6]= 26065008
>
> One may see that the matrix "t" takes less than 26Mb of memory. Now we
> try
> to import the file "temp.dat":
>
> In[1]:= MemoryInUse[]
> SetDirectory["C:/"]
> t = << temp.dat;
> MemoryInUse[]
> Out[1]= 10039112
> Out[2]= "C:\\"
> Out[4]= 86058504
>
> The same data takes more than 80Mb of memory! How this happens?
>
> The same code in Mathematica 5.2 works as expected:
>
> In[1]:=
> MemoryInUse[]
> t=RandomReal[{-1,1},{1000000,2}];
> MemoryInUse[]
> SetDirectory["C:/"]
> t>>temp.dat
> MemoryInUse[]
> Pause[2]
> Quit[]
>
> Out[1]=
> 2128352
>
> Out[3]=
> 2129928
>
> Out[4]=
> C:\
>
> Out[6]=
> 2134576
>
> In[1]:=
> MemoryInUse[]
> SetDirectory["C:/"]
> t=<<temp.dat;
> MemoryInUse[]
>
> Out[1]=
> 2127256
>
> Out[2]=
> C:\
>
> Out[4]=
> 2129344
>
> This memory leak is critical for me because I can not import backwards
> in
> Mathematica 7 my file exported in this way from Mathematica 7. The
> file size
> is 512Mb and it may be imported in Mathematica 5.2 without
> difficulties.
>
> Any ideas?
I find that in v7 RandomReal gives a packed array, while Get does not.
I don't have v5.2 here, so I'll take your observation and say that in
this case its Get returns a packed array. Of course, in v7 you can
repack the data yourself, but unfortunately this doesn't eliminate the
memory spike from the transient unpacked phase.
Have you considered the WDX file format? It returns packed arrays (for
this case).
Vince Virgilio