Re: RandomReplacement
- To: mathgroup at smc.vnet.net
- Subject: [mg51669] Re: RandomReplacement
- From: Andrzej Kozlowski <akoz at mimuw.edu.pl>
- Date: Wed, 27 Oct 2004 23:44:58 -0400 (EDT)
- References: <20041028000310.41708.qmail@web60508.mail.yahoo.com>
- Sender: owner-wri-mathgroup at wolfram.com
On 28 Oct 2004, at 09:03, sean kim wrote: > > hi andrej and everyone > > I'm starting to think *maybe* the PackedArray isn't > the problem. To reproduce the problem... Wel, first you have to make clear the meaning of "problem". Packed Arrays are not a problem in themselves: they are a "good thing' for they speed things up greatly. The problem is that when Mathematica uses Packed Arrays it does things differently than when it does not; using a compiled code that is impossible or at least very difficult for the user to modify. In our case the problem was that the packed array way of making a table of random reals uses the built in Random[] generator and is not affected by any changes the user makes to the global Random[] function. So in my RandomReplacement package I used a trick to temporarily suppress using packed arrays. The effect is to create a big slow down. But also note the word "temporarily"! > > In[1]:= SeedRandom[5]; > Timing[First[real1=Table[Random[],{249}]]] > Developer`PackedArrayQ[real1] > > Out[2]= {0. Second,0.786599} > Out[3]= False > > In[4]:= SeedRandom[5]; > Timing[First[real2=Table[Random[],{250}]]] > Developer`PackedArrayQ[real2] > > Out[5]= {0. Second,0.786599} > Out[6]= True > > and there it is. Yes, this illustrates the way Packed Arrays work here. You only get a packed array if you use the function Random[] in table and the table is large enough. > now... > > Quit > > In[1]:= Needs["Statistics`ContinuousDistributions`"]; > In[2]:= uni= UniformDistribution[0, 1] > Out[2]= UniformDistribution[0,1] > > In[3]:= SeedRandom[5]; > Timing[First[uni1=Table[Random[uni],{249}]]] > Developer`PackedArrayQ[uni1] > > Out[4]= {0. Second,0.786599} > Out[5]= False > > > In[6]:= SeedRandom[5]; > Timing[First[uni2=Table[Random[uni],{250}]]] > Developer`PackedArrayQ[uni2] > > Out[7]= {0.01 Second,0.786599} > Out[8]= False > > Without loading RandomReplacement, we see the > packedarray problem is no longer present if we change > way we use the Random. Indded PackedArrays are not used if you make your random table in this way. What you ge tis "statistically" equivalent ot using Random[] but not equivalent to it in the way Mathemaitca does the computation. So this way could also be used as a "workaround". But notice the speed difference between this way of making a uniformly distributed table and the Random[] in way: Needs["Statistics`ContinuousDistributions`"]; SeedRandom[5]; Timing[First[Table[Random[UniformDistribution[]],{10^6}]]] {13.55 Second,0.786599} SeedRandom[5]; Timing[First[Table[Random[],{10^6}]]] {0.63 Second,0.786599} So although one can base a workaround on this and RandomReplacement it won't significantly improve the performance over what RandomReplacement does. When you do not use random arrays the loss of performance is huge. > > Needs["Statistics`ContinuousDistributions`"]; > uni= UniformDistribution[0, 1] > Random[uni] > > should be identical to Random[] ( according to help) > > but once we load the RandomReplacement, > > Quit > > In[1]:= <<RandomReplacement.m > In[2]:= SeedRandom[5]; > Timing[First[real3=Table[Random[],{249}]]] > Developer`PackedArrayQ[real3] > > Timing[First[real4=Table[Random[],{250}]]] > Developer`PackedArrayQ[real4] > > Out[3]= {0. Second,0.66205} > Out[4]= False > Out[5]= {0. Second,0.793703} > Out[6]= True > > problem with packedarray is still there. No the "problem" is not there. I used Developer`ToPackedArray in my code to create a packed array as the final output, as would happen without the RandomReplacement package. So you you do get a packed array at the end, which should make things fast if you now perform certain computations on this output, but the PackedArray technology was suppressed during the making of the list: hence the performance hit. Unfortunately i have not found any way to significantly speed things over what RandomReplacement already does. Various tricks that I have tried produced an improvement of less than 10%, so in the end I did not bother trying to implement them. Basically any method of generating a random table that does not use the PackedArray technology will be much slower. But note that the PackedArray technology is not used if you generate random tables using any of the distributions defined in the Statistics` packages, so in such a case the loss of performance due to RandomReplacement` is much smaller. Andrzej Kozlowski Chiba, Japan http://www.akikoz.net/~andrzej/ http://www.mimuw.edu.pl/~akoz/