Memory-mapped files
- To: mathgroup at smc.vnet.net
- Subject: [mg89676] Memory-mapped files
- From: rych <rychphd at gmail.com>
- Date: Tue, 17 Jun 2008 00:39:30 -0400 (EDT)
This is a suggestion to the Mathematica developers. Another reason for me to stay with C++, besides the run-time efficiency, has been working with large datasets. In Mathematica one can work with packed numerical arrays that are memory-based. Why not take the next step and allow the memory-mapped files as the medium for the array? You get a normal-looking C-pointer, and Windows or Linux (64-bit versions helps) take care of the paging operations, which they're supposed to be good at. One would then view through a "window" into this data while testing the algorithms or functions, and then apply it the whole dataset, leaving it overnight, if necessary, (* this would either create the file and resize it, or gives the access to an existing file, by returning the List *) data = MMap(filename, type, dimensions); sample = data[[12000000 ;; 12000500]]; f[sample]; (*apply to the whole datasets, issue a warning that it may take a long time*) f[data]; All the best, Igor