Re: RandomReplacement
- To: mathgroup at smc.vnet.net
- Subject: [mg51669] Re: RandomReplacement
- From: Andrzej Kozlowski <akoz at mimuw.edu.pl>
- Date: Wed, 27 Oct 2004 23:44:58 -0400 (EDT)
- References: <20041028000310.41708.qmail@web60508.mail.yahoo.com>
- Sender: owner-wri-mathgroup at wolfram.com
On 28 Oct 2004, at 09:03, sean kim wrote:
>
> hi andrej and everyone
>
> I'm starting to think *maybe* the PackedArray isn't
> the problem. To reproduce the problem...
Wel, first you have to make clear the meaning of "problem". Packed
Arrays are not a problem in themselves: they are a "good thing' for
they speed things up greatly. The problem is that when Mathematica uses
Packed Arrays it does things differently than when it does not; using a
compiled code that is impossible or at least very difficult for the
user to modify. In our case the problem was that the packed array way
of making a table of random reals uses the built in Random[] generator
and is not affected by any changes the user makes to the global
Random[] function. So in my RandomReplacement package I used a trick to
temporarily suppress using packed arrays. The effect is to create a big
slow down. But also note the word "temporarily"!
>
> In[1]:= SeedRandom[5];
> Timing[First[real1=Table[Random[],{249}]]]
> Developer`PackedArrayQ[real1]
>
> Out[2]= {0. Second,0.786599}
> Out[3]= False
>
> In[4]:= SeedRandom[5];
> Timing[First[real2=Table[Random[],{250}]]]
> Developer`PackedArrayQ[real2]
>
> Out[5]= {0. Second,0.786599}
> Out[6]= True
>
> and there it is.
Yes, this illustrates the way Packed Arrays work here. You only get a
packed array if you use the function Random[] in table and the table is
large enough.
> now...
>
> Quit
>
> In[1]:= Needs["Statistics`ContinuousDistributions`"];
> In[2]:= uni= UniformDistribution[0, 1]
> Out[2]= UniformDistribution[0,1]
>
> In[3]:= SeedRandom[5];
> Timing[First[uni1=Table[Random[uni],{249}]]]
> Developer`PackedArrayQ[uni1]
>
> Out[4]= {0. Second,0.786599}
> Out[5]= False
>
>
> In[6]:= SeedRandom[5];
> Timing[First[uni2=Table[Random[uni],{250}]]]
> Developer`PackedArrayQ[uni2]
>
> Out[7]= {0.01 Second,0.786599}
> Out[8]= False
>
> Without loading RandomReplacement, we see the
> packedarray problem is no longer present if we change
> way we use the Random.
Indded PackedArrays are not used if you make your random table in this
way. What you ge tis "statistically" equivalent ot using Random[] but
not equivalent to it in the way Mathemaitca does the computation. So
this way could also be used as a "workaround". But notice the speed
difference between this way of making a uniformly distributed table and
the Random[] in way:
Needs["Statistics`ContinuousDistributions`"];
SeedRandom[5];
Timing[First[Table[Random[UniformDistribution[]],{10^6}]]]
{13.55 Second,0.786599}
SeedRandom[5];
Timing[First[Table[Random[],{10^6}]]]
{0.63 Second,0.786599}
So although one can base a workaround on this and RandomReplacement it
won't significantly improve the performance over what RandomReplacement
does. When you do not use random arrays the loss of performance is
huge.
>
> Needs["Statistics`ContinuousDistributions`"];
> uni= UniformDistribution[0, 1]
> Random[uni]
>
> should be identical to Random[] ( according to help)
>
> but once we load the RandomReplacement,
>
> Quit
>
> In[1]:= <<RandomReplacement.m
> In[2]:= SeedRandom[5];
> Timing[First[real3=Table[Random[],{249}]]]
> Developer`PackedArrayQ[real3]
>
> Timing[First[real4=Table[Random[],{250}]]]
> Developer`PackedArrayQ[real4]
>
> Out[3]= {0. Second,0.66205}
> Out[4]= False
> Out[5]= {0. Second,0.793703}
> Out[6]= True
>
> problem with packedarray is still there.
No the "problem" is not there. I used Developer`ToPackedArray in my
code to create a packed array as the final output, as would happen
without the RandomReplacement package. So you you do get a packed array
at the end, which should make things fast if you now perform certain
computations on this output, but the PackedArray technology was
suppressed during the making of the list: hence the performance hit.
Unfortunately i have not found any way to significantly speed things
over what RandomReplacement already does. Various tricks that I have
tried produced an improvement of less than 10%, so in the end I did not
bother trying to implement them. Basically any method of generating a
random table that does not use the PackedArray technology will be much
slower. But note that the PackedArray technology is not used if you
generate random tables using any of the distributions defined in the
Statistics` packages, so in such a case the loss of performance due to
RandomReplacement` is much smaller.
Andrzej Kozlowski
Chiba, Japan
http://www.akikoz.net/~andrzej/
http://www.mimuw.edu.pl/~akoz/