Efficient way of reading matrices
- To: mathgroup at smc.vnet.net
- Subject: [mg85229] Efficient way of reading matrices
- From: Szabolcs Horvát <szhorvat at gmail.com>
- Date: Sat, 2 Feb 2008 23:32:25 -0500 (EST)
I need to read in some very big integer matrices from text files, the biggest of which have a dimension of approx 200*250000. What is the most efficient way of doing this? The problem is that ReadList[#, Number, RecordLists->True] cannot do this without swapping (it takes more than 10 minutes to finish), but after the matrix has been read in and compacted with ToPackedArray, it only takes up 200 MBs, and I can work with it comfortably. Are there more efficient ways to read in such big files? Is there any way to tell Mathematica that all the integers are machine-size and the matrix is rectangular (i.e. not a ragged array)? A last resort would be to try to write a MathLink program for reading the data but I would like to avoid this ...