Re: Conditions with Statistical Functions

*To*: mathgroup at smc.vnet.net*Subject*: [mg66163] Re: Conditions with Statistical Functions*From*: Gregory Lypny <gregory.lypny at videotron.ca>*Date*: Tue, 2 May 2006 02:43:19 -0400 (EDT)*References*: <16568608.1146308002406.JavaMail.root@eastrmwml01.mgt.cox.net> <e31vb8$hgd$1@smc.vnet.net> <4454EC45.8030009@gmail.com> <B242D500-B1BF-4293-BB70-3C9C562AFF4D@videotron.ca> <22d35c5a0605010846p7c08eb61y26e91da12bbeae7d@mail.gmail.com>*Sender*: owner-wri-mathgroup at wolfram.com

Hi Jean-Marc, This has been a big help! Thank you. It was very kind of you to take the time to write such a lucid and detailed explanation. I'm embarrassed about the transposition question. That had occurred to me when I started thinking about the problem initially, but I quickly forgot about it when I got bogged down in trying to construct the appropriate function (unsuccessfully, of course). I think your explanation has helped me better understand the use of slots (#) and the ampersand (&). If I understand correctly, the ampersand is part of the definition of a pure function and is therefore required to terminate it. That means that its placement in nested functions is important. So, at the risk of messing up, the first ampersand turns the greater-than condition into a pure function, and the second turns the Select part, along with the enclosing greater-than condition, into another pure function. Each has its own slot (#1) or user-supplied variable, which in your example is the matrix "1st". I will now apply it to my data. I suspect I will be able to cut down processing time over using a Do loop. If I'm feeling courageous, I will try to use your approach and Bob Hanlon's to create similar functions using Count. Regards, Greg On Mon, May 1, 2006, at 11:46 AM, Jean-Marc Gulliet wrote: > On 5/1/06, Gregory Lypny <gregory.lypny at videotron.ca> wrote: >> Thanks again, Jean-Marc. >> >> This is elegant. As I mentioned before, I'm facing an uphill >> battle with >> Mathematica's syntax, and in particular, the meaning and placement >> of things >> like #, #1, or #[[1]], &, @, @@, @@@, /@, /. . It's all a bit >> overwhelming! >> >> I think I understand your second version better, so I'll work with >> it first. >> If I'm not mistaken, /@ tells the Mean function to map over its >> argument >> and #1 directs that mapping to the first argument, which for Mean >> applied to >> a matrix is a vector. What I never would have gotten on my own is >> the use >> of the second ampersand in parenthesis. Is that meant to connect >> Select >> with Transpose? I'm also not quite clear on why we need Transpose >> because >> Mean operates on columns anyway. >> >> Regards, >> >> Greg >> >> >> >> On Sun, Apr 30, 2006, at 12:56 PM, Jean-Marc Gulliet wrote: >> >> >> In[3]:= >> >> Mean /@ (Transpose[lst /. x_ /; x <= 100 -> 0] /. >> >> 0 -> Sequence[]) >> >> >> >> >> Out[3]= >> >> 1311 1450 527 >> >> {125, ----, ----, ---} >> >> 8 9 3 >> >> >> >> >> In[4]:= >> >> Mean /@ (Select[#1, #1 > 100 & ] & ) /@ Transpose[lst] >> >> >> >> >> Out[4]= >> >> 1311 1450 527 >> >> {125, ----, ----, ---} >> >> 8 9 3 >> >> >> >> >> Best regards, >> >> Jean-Marc >> > Hi Gregory, > > Let us try to follow gradually what is going on. Say that our data set > is a 12 by 4 matrix of integer entries: > > lst={{109,168,173,109},{4,143,200,90},{181,162,85,196},{30, > 108,86,34},{94,127,144,34},{199,109,195,188},{176, > 34,46,110},{95,27,160,109},{43, > 71,130,66},{56,148,109,163},{110,43,50,53},{32,34,16,95}} > > First, we will focus on the second expression, which use the > *Select* function. > > Mean/@(Select[#1,#1>100&]&)/@Transpose[lst] > > Out[3]= > 965 1111 875 > {155, ---, ----, ---} > 7 7 6 > > If we look at the full form ? Mathematica internal representation ? of > the above expression, we see that (Select[#1, #1 > 100 & ] & ) is an > expression made of nested pure functions. Pure functions are > equivalent to anonymous functions in, say, LISP. The #n's (or > Slot[n]'s) are placeholders for variables and a pure function > definition ends by an ampersand character '&'. In our function, it is > really important to realize that the first #1 has nothing to do with > the second #1 because they are located in different function > definitions (this is clearer in the full form of the expression). > > In[6]:= > FullForm[HoldForm[Mean /@ (Select[#1, #1 > 100 & ] & ) /@ Transpose > [lst]]] > > Out[6]//FullForm= > HoldForm[Map[Mean, > Map[Function[Select[Slot[ > 1],Function[Greater[Slot[1],100]]]],Transpose[lst]]]] > > Note, we could have written the pure function with explicit ? local > ?variable name such as in the line below. *Select* works on each row > and applies the test to each element. > > In[10]:= > Map[Mean,Map[Function[rowvec,Select[rowvec,Function[elem, > Greater[elem,100]]]],Transpose[lst]]] > > Out[10]= > 965 1111 875 > {155, ---, ----, ---} > 7 7 6 > > Now, let us see why we need to transpose the matrix first. Below, we > can see the original matrix. By visual inspection, we notice that the > first column has five values that are greater than 100, and the > second, third and fourth columns have seven,seven,and six values > greater than 100, respectively. > > In[2]:= > TableForm[lst] > > Out[2]//TableForm= > 109 168 173 109 > > 4 143 200 90 > > 181 162 85 196 > > 30 108 86 34 > > 94 127 144 34 > > 199 109 195 188 > > 176 34 46 110 > > 95 27 160 109 > > 43 71 130 66 > > 56 148 109 163 > > 110 43 50 53 > > 32 34 16 95 > > Applying the select clause to the original matrix without > transposition yields the following result: any values less than or > equal to 100 have been discarded; however, the structure of the > original matrix has been changed too. Not only do we have a collection > of row vectors of unequal lengths, but also many elements have shifted > to the left! > > In[13]:= > (Select[#1,#1>100&]&)/@lst//TableForm > > Out[13]//TableForm= > 109 168 173 109 > > 143 200 > > 181 162 196 > > 108 > > 127 144 > > 199 109 195 188 > > 176 110 > > 160 109 > > 130 > > 148 109 163 > > 110 > On the other hand, transposing first allows getting the correct values > in each row. > > In[12]:= > (Select[#1,#1>100&]&)/@Transpose[lst]//TableForm > > Out[12]//TableForm= > 109 181 199 176 110 > > 168 143 162 108 127 109 148 > > 173 200 144 195 160 130 109 > > 109 196 188 110 109 163 > > Finally, mapping *Mean* to the resulting list of lists allows to > compute the mean of each row vectors that correspond to our original > columns without the unwanted values (so using *Map* allows us to > change the behavior of Mean which is to compute column by column). > > In[15]:= > Mean/@(Select[#1,#1>100&]&)/@Transpose[lst]//Trace > > Well, I hope that I have not been to long and to obscure in my > explanation and that indeed I have successfully shred some light on > Mathematica programming. > > Best regards, > Jean-Marc