Large-scale enumerations and memory problems
- To: mathgroup at smc.vnet.net
- Subject: [mg35084] Large-scale enumerations and memory problems
- From: "Mark E. Berres" <markb at ravel.zoology.wisc.edu>
- Date: Mon, 24 Jun 2002 03:21:36 -0400 (EDT)
- Organization: University of Wisconsin, Madison
- Sender: owner-wri-mathgroup at wolfram.com
Hi,
I am performing some enumerations (v4.1 under Win-XP with 768 MB
physical memory and
4GB swap space) with the following simple code:
Enum[l_List, (k_Integer)?Positive] := Block[{minusone = Enum[l, k - 1], i,
n = Length[l]}, Join @@ Table[(Prepend[#1, l[[i]]] & ) /@ minusone, {i,
n}]].
What this does is to take a supplied list of objects, l, and generate all
possible combinations of
these objects in a string of length k. Thus, even for small l and moderate
values of k, the number
of combinations becomes very large.
I am exporting the results to a file by
Export["file.txt", Enum[{G , A , T, C}, 6], "List"] (I want one entry per
line of text)
but when k becomes "too large", for example if l=4
and k=14, 4^14 = ~268 million, the kernal eventually crashes complaining
about memory. I estimated
that approx. 1.61 GB is required to store all strings in memory before
writing to disk - plenty of room
available on my machine, but still it crashes. It appears that the
calculation first does its thing and then
when initiating a write to disk, crashes. Is there a way to write the
results one-at-a-time, rather than
the whole-thing-at-once? Or does my problem lie elsewhere?
Sincerely,
Mark