Re: ByteCount of imported machine-precision data matrix three times
- To: mathgroup at smc.vnet.net
- Subject: [mg92209] Re: ByteCount of imported machine-precision data matrix three times
- From: dh <dh at metrohm.ch>
- Date: Tue, 23 Sep 2008 07:31:23 -0400 (EDT)
- References: <gb7odj$nij$1@smc.vnet.net>
Hi Gareth,
I can not explain what is hapening, but it has nothing to do with your
file. Maybe Wolfram could give some insight. Consider:
dummyData=10^10 Table[ RandomReal[{-1,1}],{20},{14}];
ByteCount[dummyData]
dummyData=10^10 Table[10^10 RandomReal[{-1,1}],{20},{14}];
ByteCount[dummyData]
Daniel
Gareth Russell wrote:
> Hi,
>
> I am encountering some strange memory-related behavior when importing
> numerical data from a file. If anyone is interested, a (small) example
> file is here:
>
> http://web.njit.edu/~russell/Mathematica.html
>
> It's a simple 2D array of numbers. The issue is that when imported,
> ByteCount[] indicates that the resultant expression takes up more than
> three times as much memory as an equivalent machine-precision matrix
> generated within Mathematica. All diagnostics that I can think of
> indicate that the imported expression is equivalent in precision. And
> indeed, ByteCount applied to individual elements of each matrix returns
> 16 as an answer. It's only the overall ByteCount which is hugely
> different.
>
> I discovered a workaround: if I generate a dummy matrix of 0. elements
> (which has the smaller ByteCount), and add it to the imported matrix,
> the result, while appearing identical (as it should), now also has the
> smaller ByteCount.
>
> Does anyone know what it going on here? Until I discovered the
> workaround it was a problem, as I need to read in a large number of
> much larger matrices all together, was encountering memory issues.
>
> Thanks,
>
> Gareth
> NJIT
>
>
>
> In[1452]:= data = Drop[Import["12e.dat"], 6];
>
> In[1453]:= Dimensions[data]
>
> Out[1453]= {20, 14}
>
> In[1454]:= {Min[Flatten[data]], Max[Flatten[data]]}
>
> Out[1454]= {-1.1, 0.992655}
>
> In[1455]:= MachineNumberQ[data[[1, 1]]]
>
> Out[1455]= True
>
> In[1456]:= ByteCount[data]
>
> Out[1456]= 7384
>
> In[1466]:= ByteCount[data[[1, 1]]]
>
> Out[1466]= 16
>
> In[1457]:= dummyData = Table[RandomReal[{-1.1, 1}], {20}, {14}];
>
> In[1458]:= ByteCount[dummyData]
>
> Out[1458]= 2360
>
> In[1460]:= zeroData = Table[0., {20}, {14}];
>
> In[1461]:= ByteCount[zeroData]
>
> Out[1461]= 2360
>
> In[1462]:= newData = data + zeroData;
>
> In[1463]:= {Min[Flatten[newData]], Max[Flatten[newData]]}
>
> Out[1463]= {-1.1, 0.992655}
>
> In[1464]:= MachineNumberQ[newData[[1, 1]]]
>
> Out[1464]= True
>
> In[1465]:= ByteCount[newData]
>
> Out[1465]= 2360
>
> In[1467]:= ByteCount[newData[[1, 1]]]
>
> Out[1467]= 16
>
--
Daniel Huber
Metrohm Ltd.
Oberdorfstr. 68
CH-9100 Herisau
Tel. +41 71 353 8585, Fax +41 71 353 8907
E-Mail:<mailto:dh at metrohm.com>
Internet:<http://www.metrohm.com>