Efficient way of reading matrices
- To: mathgroup at smc.vnet.net
- Subject: [mg85229] Efficient way of reading matrices
- From: Szabolcs Horvát <szhorvat at gmail.com>
- Date: Sat, 2 Feb 2008 23:32:25 -0500 (EST)
I need to read in some very big integer matrices from text files, the
biggest of which have a dimension of approx 200*250000.
What is the most efficient way of doing this?
The problem is that ReadList[#, Number, RecordLists->True] cannot do
this without swapping (it takes more than 10 minutes to finish), but
after the matrix has been read in and compacted with ToPackedArray, it
only takes up 200 MBs, and I can work with it comfortably.
Are there more efficient ways to read in such big files? Is there any
way to tell Mathematica that all the integers are machine-size and the
matrix is rectangular (i.e. not a ragged array)?
A last resort would be to try to write a MathLink program for reading
the data but I would like to avoid this ...
Prev by Date:
Re: For Loop and NDSolve
Next by Date:
Re: Convex random polyhedrons
Previous by thread:
NONE of the Mathematica versions is able to calculate the integrals
Next by thread:
Re: Efficient way of reading matrices