Re: Efficient way of reading matrices

*To*: mathgroup at smc.vnet.net*Subject*: [mg85237] Re: [mg85229] Efficient way of reading matrices*From*: "Szabolcs HorvÃt" <szhorvat at gmail.com>*Date*: Mon, 4 Feb 2008 03:06:09 -0500 (EST)*References*: <200802030432.XAA11313@smc.vnet.net>

On Feb 3, 2008 8:13 PM, Sseziwa Mukasa <mukasa at jeol.com> wrote: > > On Feb 2, 2008, at 11:32 PM, Szabolcs HorvÃ¡t wrote: > > > > > I need to read in some very big integer matrices from text files, the > > biggest of which have a dimension of approx 200*250000. > > > > What is the most efficient way of doing this? > > > > The problem is that ReadList[#, Number, RecordLists->True] cannot do > > this without swapping (it takes more than 10 minutes to finish), but > > after the matrix has been read in and compacted with ToPackedArray, it > > only takes up 200 MBs, and I can work with it comfortably. > > > > Are there more efficient ways to read in such big files? Is there any > > way to tell Mathematica that all the integers are machine-size and the > > matrix is rectangular (i.e. not a ragged array)? > > > > A last resort would be to try to write a MathLink program for reading > > the data but I would like to avoid this ... > > If you are reading machine numbers then BinaryReadList seems more > appropriate. By "machine-size number" I meant that the numbers are guaranteed to fit into 32 bits, therefore they can be stored in a packed array. Mathematica can deal with numbers of arbitrary size, and the fact that it does not know beforehand how big the numbers in the file are going to be is probably one of the reasons why it uses so much memory while reading the file.

**References**:**Efficient way of reading matrices***From:*Szabolcs Horvát <szhorvat@gmail.com>