MathGroup Archive 2009

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Huge memory leaks when importing two-dimensional array in Mathematica

  • To: mathgroup at smc.vnet.net
  • Subject: [mg103463] Re: Huge memory leaks when importing two-dimensional array in Mathematica
  • From: "Alexey Popkov" <lehin.p at gmail.com>
  • Date: Wed, 23 Sep 2009 07:24:37 -0400 (EDT)
  • References: <h9927o$1nm$1@smc.vnet.net>
  • Reply-to: "Alexey Popkov" <lehin.p at gmail.com>

I apologize because the code I have posted here for Mathematica 5.2 was 
wrong. The real behavior in v.5.2 is as follows:

In[1]:=
MemoryInUse[]
t = Table[Random[Real, {-1, 1}], {1000000}, {2}];
MemoryInUse[]
SetDirectory["C:/"]
t >> temp.dat
MemoryInUse[]
MaxMemoryUsed[]
Pause[3]
Quit[]

Out[1]=2128760
Out[3]=18132392
Out[4]=C:\
Out[6]=18138208
Out[7]=91137872


In[1]:=MemoryInUse[]
SetDirectory["C:/"]
t = << temp.dat;
MemoryInUse[]
MaxMemoryUsed[]
Out[1]=2127368
Out[2]=C:\
Out[4]=70129064
Out[5]=120570008

So in really v.5.2 behaves like v.7.01 in this case. But nevertheless I can 
import my two-dimensional array (file "2DArray1.dat" with size 554 482 863 
bytes) only in v.5.2 but not in v.7.01. Interestingly, another 2D-array 
(file "2DArray2.dat" with size 454 468 512) is imported in v.7 without 
difficulties. The comparison follows.

In v.5.2:
In[1]:=
MemoryInUse[]
SetDirectory["C:/"]
t = << "2DArray1.dat";
MemoryInUse[]
MaxMemoryUsed[]
Out[1]=2127432
Out[2]=C:\
Out[4]=818129176
Out[5]=1463533184

In v.7.01:

In[1]:= MemoryInUse[]
SetDirectory["C:/"]
t = << "2DArray1.dat";
MemoryInUse[]
MaxMemoryUsed[]
Out[1]= 10039832
Out[2]= "C:\\"
During evaluation of In[1]:=
No more memory available.
Mathematica kernel has shut down.
Try quitting other applications and then retry.


Another (smaller) array:

In v.5.2:

In[1]:=
MemoryInUse[]
SetDirectory["C:/"]
t = << "2DArray2.dat";
MemoryInUse[]
MaxMemoryUsed[]
Out[1]=2127432
Out[2]=C:\
Out[4]=682129192
Out[5]=1237539808

In v.7.01:

In[1]:= MemoryInUse[]
SetDirectory["C:/"]
t = << "2DArray2.dat";
MemoryInUse[]
MaxMemoryUsed[]
Out[1]= 10039832
Out[2]= "C:\\"
Out[4]= 770058936
Out[5]= 1291636032



----- Original Message ----- 
From: "Alexey" <lehin.p at gmail.com>
Newsgroups: comp.soft-sys.math.mathematica
Sent: Tuesday, September 22, 2009 3:28 AM
Subject: [mg103463] Huge memory leaks when importing two-dimensional array in 
Mathematica


> Hello,
> Consider the following (evaluated in Mathematica 7.01):
>
> In[1]:= MemoryInUse[]
> t = RandomReal[{-1, 1}, {1000000, 2}];
> MemoryInUse[]
> SetDirectory["C:/"]
> t >> temp.dat
> MemoryInUse[]
> Quit[]
> Out[1]= 10040344
> Out[3]= 26058000
> Out[4]= "C:\\"
> Out[6]= 26065008
>
> One may see that the matrix "t" takes less than 26Mb of memory. Now we
> try
> to import the file "temp.dat":
>
> In[1]:= MemoryInUse[]
> SetDirectory["C:/"]
> t = << temp.dat;
> MemoryInUse[]
> Out[1]= 10039112
> Out[2]= "C:\\"
> Out[4]= 86058504
>
> The same data takes more than 80Mb of memory! How this happens?
>
> The same code in Mathematica 5.2 works as expected:
>
> In[1]:=
> MemoryInUse[]
> t=RandomReal[{-1,1},{1000000,2}];
> MemoryInUse[]
> SetDirectory["C:/"]
> t>>temp.dat
> MemoryInUse[]
> Pause[2]
> Quit[]
>
> Out[1]=
> 2128352
>
> Out[3]=
> 2129928
>
> Out[4]=
> C:\
>
> Out[6]=
> 2134576
>
> In[1]:=
> MemoryInUse[]
> SetDirectory["C:/"]
> t=<<temp.dat;
> MemoryInUse[]
>
> Out[1]=
> 2127256
>
> Out[2]=
> C:\
>
> Out[4]=
> 2129344
>
> This memory leak is critical for me because I can not import backwards
> in
> Mathematica 7 my file exported in this way from Mathematica 7. The
> file size
> is 512Mb and it may be imported in Mathematica 5.2 without
> difficulties.
>
> Any ideas?
> 



  • Prev by Date: Re: Convert Hue[] to RGBColor[] and backwards
  • Next by Date: Showing the Current Value of a Slider in Manipulate
  • Previous by thread: Huge memory leaks when importing two-dimensional array in Mathematica
  • Next by thread: Failover and high availability