Re: Plotting a large dataset
- To: mathgroup at smc.vnet.net
- Subject: [mg94255] Re: Plotting a large dataset
- From: Jean-Marc Gulliet <jeanmarc.gulliet at gmail.com>
- Date: Tue, 9 Dec 2008 06:57:55 -0500 (EST)
- Organization: The Open University, Milton Keynes, UK
- References: <ghj057$fid$1@smc.vnet.net>
Nikolaus Rath wrote:
> I have trouble plotting a dataset of 2048000 reals. I tried the
> following code (as recently suggested on the list):
>
> xdata1 = ReadList[prefix <> "1.dat", Real, RecordLists -> False];
> xdata1 = Developer`ToPackedArray[xdata1];
> Dimensions[xdata1]
> (* yields {2048000} *)
> ListPlot[xdata1, PerformanceGoal -> "Speed"]
>
>
> but I'm simply running out of main memory (2 GB available). Since I
> can plot the same data within seconds using IDL, I guess Mathematica
> should be able to do the same. But how?
Hi Nikolaus,
Assuming I have understood correctly what the problem is, using the
option *MaxPlotPoints*, fed with some appropriate value depending on
your actual dataset, should allow ListPlot[] to display a meaningful
plot exhibiting the main characteristics or trends of your data while
being memory conservative.
(I am not familiar enough with IDL to be hundred percent positive here,
but I doubt that it plots all the points: a vector of two million points
to be displayed is likely to result in some patch of solid color on a
white canvas...)
Note that even though your imported data are stored as a packed array,
this does not mean nor guarantee that subsequent computations are going
to work directly with such structures. Indeed, as you can see in the
code below, ListPlot[] unpacks xdata1 at some point to work on it, then
released the memory.
One can check when packed arrays are unpacked by setting the option
"UnpackMessage" to True as in
SetSystemOptions[ "PackedArrayOptions" -> {"UnpackMessage" -> True}]
The current amount of memory used and the maximum amount of memory that
was required at any given time during a Mathematica session can be
displayed thanks to the functions MemoryInUse[] and MaxMemoryUsed[],
respectively.
Setting the system variable $HistoryLength to zero preserves memory by
not storing the results of the evaluations in the variables Out[xx].
In[1]:=
$HistoryLength = 0; Needs["Developer`"]; $Version
SetSystemOptions[ "PackedArrayOptions" -> {"UnpackMessage" -> True}];
{MemoryInUse[], MaxMemoryUsed[]}
numPts = 2048000;
xdata1 = Table[N@Sin[n], {n, 0, numPts - 1}];
If[PackedArrayQ[xdata1], , xdata1 = ToPackedArray[xdata1]];
If[PackedArrayQ[xdata1], PackedArrayForm[xdata1]]
{MemoryInUse[], MaxMemoryUsed[]}
p = ListPlot[xdata1, PerformanceGoal -> "Speed",
MaxPlotPoints -> numPts/10]
{ByteCount[p], MemoryInUse[], MaxMemoryUsed[]}
{ByteCount[xdata1], ByteCount[FromPackedArray[xdata1]]}
Out[1]= "6.0 for Mac OS X x86 (64-bit) (May 21, 2008)"
Out[3]= {11169864, 11557560}
Out[7]= "PackedArray"[Real, "<"2048000">"]
Out[8]= {27554192, 27557416}
During evaluation of In[1]:= FromPackedArray::unpack1: Unpacking \
array. >>
Out[10]= {3278016, 38398648, 437040856}
During evaluation of In[1]:= FromPackedArray::unpack1: Unpacking \
array. >>
Out[11]= {16384116, 49152032}
Hope this helps,
-- Jean-Marc