Services & Resources / Wolfram Forums / MathGroup Archive
-----

MathGroup Archive 2008

[Date Index] [Thread Index] [Author Index]

Search the Archive

Memory-mapped files

  • To: mathgroup at smc.vnet.net
  • Subject: [mg89676] Memory-mapped files
  • From: rych <rychphd at gmail.com>
  • Date: Tue, 17 Jun 2008 00:39:30 -0400 (EDT)

This is a suggestion to the Mathematica developers.
Another reason for me to stay with C++, besides the run-time
efficiency, has been working with large datasets. In Mathematica one
can work with packed numerical arrays that are memory-based. Why not
take the next step and allow the memory-mapped files as the medium for
the array? You get a normal-looking C-pointer, and Windows or Linux
(64-bit versions helps) take care of the paging operations, which
they're supposed to be good at.

One would then view through a "window" into this data while testing
the algorithms or functions, and then apply it the whole dataset,
leaving it overnight, if necessary,

(* this would either create the file and resize it, or gives the
access to an existing file, by returning the List *)
data = MMap(filename, type, dimensions);
sample = data[[12000000 ;; 12000500]];
f[sample];
(*apply to the whole datasets, issue a warning that it may take a long
time*)
f[data];

All the best,
Igor


  • Prev by Date: Re: Problems with FindRoot and recursive functions
  • Next by Date: Re: Workbench editing snag
  • Previous by thread: Mathematica as an OPC Client
  • Next by thread: Power Tower and Carmichael Lambda Function