[Date Index] [Thread Index] [Author Index]

Efficient way of reading matrices

• To: mathgroup at smc.vnet.net
• Subject: [mg85229] Efficient way of reading matrices
• From: Szabolcs Horvát <szhorvat at gmail.com>
• Date: Sat, 2 Feb 2008 23:32:25 -0500 (EST)

I need to read in some very big integer matrices from text files, the 
biggest of which have a dimension of approx 200*250000.

What is the most efficient way of doing this?

The problem is that ReadList[#, Number, RecordLists->True] cannot do 
this without swapping (it takes more than 10 minutes to finish), but 
after the matrix has been read in and compacted with ToPackedArray, it 
only takes up 200 MBs, and I can work with it comfortably.

Are there more efficient ways to read in such big files?  Is there any 
way to tell Mathematica that all the integers are machine-size and the 
matrix is rectangular (i.e. not a ragged array)?

A last resort would be to try to write a MathLink program for reading 
the data but I would like to avoid this ...

• Follow-Ups:
  • Re: Efficient way of reading matrices
    • From: "Szabolcs HorvÃt" <szhorvat@gmail.com>
  • Re: Efficient way of reading matrices
    • From: Sseziwa Mukasa <mukasa@jeol.com>