[Date Index] [Thread Index] [Author Index]

ByteCount of imported machine-precision data matrix three times equivalent generated matrix

• To: mathgroup at smc.vnet.net
• Subject: [mg92171] ByteCount of imported machine-precision data matrix three times equivalent generated matrix
• From: Gareth Russell <russell at njit.edu>
• Date: Mon, 22 Sep 2008 05:27:24 -0400 (EDT)

Hi,

I am encountering some strange memory-related behavior when importing 
numerical data from a file. If anyone is interested, a (small) example 
file is here:

http://web.njit.edu/~russell/Mathematica.html

It's a simple 2D array of numbers. The issue is that when imported, 
ByteCount[] indicates that the resultant expression takes up more than 
three times as much memory as an equivalent machine-precision matrix 
generated within Mathematica. All diagnostics that I can think of 
indicate that the imported expression is equivalent in precision. And 
indeed, ByteCount applied to individual elements of each matrix returns 
16 as an answer. It's only the overall ByteCount which is hugely 
different.

I discovered a workaround: if I generate a dummy matrix of 0. elements 
(which has the smaller ByteCount), and add it to the imported matrix, 
the result, while appearing identical (as it should), now also has the 
smaller ByteCount.

Does anyone know what it going on here? Until I discovered the 
workaround it was a problem, as I need to read in a large number of 
much larger matrices all together, was encountering memory issues.

Thanks,

Gareth
NJIT



In[1452]:= data = Drop[Import["12e.dat"], 6];

In[1453]:= Dimensions[data]

Out[1453]= {20, 14}

In[1454]:= {Min[Flatten[data]], Max[Flatten[data]]}

Out[1454]= {-1.1, 0.992655}

In[1455]:= MachineNumberQ[data[[1, 1]]]

Out[1455]= True

In[1456]:= ByteCount[data]

Out[1456]= 7384

In[1466]:= ByteCount[data[[1, 1]]]

Out[1466]= 16

In[1457]:= dummyData = Table[RandomReal[{-1.1, 1}], {20}, {14}];

In[1458]:= ByteCount[dummyData]

Out[1458]= 2360

In[1460]:= zeroData = Table[0., {20}, {14}];

In[1461]:= ByteCount[zeroData]

Out[1461]= 2360

In[1462]:= newData = data + zeroData;

In[1463]:= {Min[Flatten[newData]], Max[Flatten[newData]]}

Out[1463]= {-1.1, 0.992655}

In[1464]:= MachineNumberQ[newData[[1, 1]]]

Out[1464]= True

In[1465]:= ByteCount[newData]

Out[1465]= 2360

In[1467]:= ByteCount[newData[[1, 1]]]

Out[1467]= 16

-- 
Gareth Russell
NJIT