Re: Efficient way of reading matrices

*To*: mathgroup at smc.vnet.net*Subject*: [mg85243] Re: [mg85229] Efficient way of reading matrices*From*: Sseziwa Mukasa <mukasa at jeol.com>*Date*: Mon, 4 Feb 2008 03:09:14 -0500 (EST)*References*: <200802030432.XAA11313@smc.vnet.net>

On Feb 2, 2008, at 11:32 PM, Szabolcs Horv=E1t wrote: > > I need to read in some very big integer matrices from text files, the > biggest of which have a dimension of approx 200*250000. > > What is the most efficient way of doing this? > > The problem is that ReadList[#, Number, RecordLists->True] cannot do > this without swapping (it takes more than 10 minutes to finish), but > after the matrix has been read in and compacted with ToPackedArray, it > only takes up 200 MBs, and I can work with it comfortably. > > Are there more efficient ways to read in such big files? Is there any > way to tell Mathematica that all the integers are machine-size and the > matrix is rectangular (i.e. not a ragged array)? > > A last resort would be to try to write a MathLink program for reading > the data but I would like to avoid this ... If you are reading machine numbers then BinaryReadList seems more appropriate.

**References**:**Efficient way of reading matrices***From:*Szabolcs Horvát <szhorvat@gmail.com>