[Date Index] [Thread Index] [Author Index]
Re: Efficient way of reading matrices
On Feb 2, 2008, at 11:32 PM, Szabolcs Horv=E1t wrote: > > I need to read in some very big integer matrices from text files, the > biggest of which have a dimension of approx 200*250000. > > What is the most efficient way of doing this? > > The problem is that ReadList[#, Number, RecordLists->True] cannot do > this without swapping (it takes more than 10 minutes to finish), but > after the matrix has been read in and compacted with ToPackedArray, it > only takes up 200 MBs, and I can work with it comfortably. > > Are there more efficient ways to read in such big files? Is there any > way to tell Mathematica that all the integers are machine-size and the > matrix is rectangular (i.e. not a ragged array)? > > A last resort would be to try to write a MathLink program for reading > the data but I would like to avoid this ... If you are reading machine numbers then BinaryReadList seems more appropriate.