[Date Index] [Thread Index] [Author Index]

Re: ByteCount of imported machine-precision data matrix three times

• To: mathgroup at smc.vnet.net
• Subject: [mg92205] Re: ByteCount of imported machine-precision data matrix three times
• From: Jean-Marc Gulliet <jeanmarc.gulliet at gmail.com>
• Date: Tue, 23 Sep 2008 07:30:38 -0400 (EDT)
• Organization: The Open University, Milton Keynes, UK
• References: <gb7odj$nij$1@smc.vnet.net>

Gareth Russell wrote:

> I am encountering some strange memory-related behavior when importing 
> numerical data from a file. If anyone is interested, a (small) example 
> file is here:
> 
> http://web.njit.edu/~russell/Mathematica.html
> 
> It's a simple 2D array of numbers. The issue is that when imported, 
> ByteCount[] indicates that the resultant expression takes up more than 
> three times as much memory as an equivalent machine-precision matrix 
> generated within Mathematica. All diagnostics that I can think of 
> indicate that the imported expression is equivalent in precision. And 
> indeed, ByteCount applied to individual elements of each matrix returns 
> 16 as an answer. It's only the overall ByteCount which is hugely 
> different.

*snip*

You will get the most compact form only when your data are made of 
numeric values of same type (say, all machine integers or all 
floating-point numbers) which is not the case for your imported dataset 
(testing the first element only is not enough, see below). If it were 
the case, Mathematica would use a packed array representation [1].

Something along the line

data = Developer`ToPackedArray[
    N[Drop[Import["http://web.njit.edu/~russell/Downloads/12e.dat";],
      6]]];

will do what you want.

Here is a step by step illustration of what is going on before and after
packed arrays are used.

In[1]:= data =
   Drop[Import["http://web.njit.edu/~russell/Downloads/12e.dat";], 6];

In[2]:= Dimensions[data]

Out[2]= {20, 14}

In[3]:= ByteCount[data]

Out[3]= 7384

In[4]:= Developer`PackedArrayQ[data]

Out[4]= False

In[5]:= MatrixQ[data, MachineNumberQ]

Out[5]= False

In[6]:= data = Developer`ToPackedArray[N[data]];

In[7]:= ByteCount[data]

Out[7]= 2360

In[8]:= Developer`PackedArrayQ[data]

Out[8]= True

In[9]:= MatrixQ[data, MachineNumberQ]

Out[9]= True

Regards,
- Jean-Marc

[1] /Performance of Linear Algebra Computation/
http://reference.wolfram.com/mathematica/tutorial/LinearAlgebraPerformance.html