Re: Differentiation ?
- To: mathgroup at smc.vnet.net
- Subject: [mg14703] Re: [mg14668] Differentiation ?
- From: Jurgen Tischer <jtischer at col2.telecom.com.co>
- Date: Tue, 10 Nov 1998 01:20:57 -0500
- Organization: Universidad del Valle
- References: <004d01be0b2c$71a4e0d0$0100a8c0@bertha.bureau>
- Sender: owner-wri-mathgroup at wolfram.com
Well Yves,
I don't think math works different in statistics, so let me try once
more.
You start with
e'e=(y-X b)'(y-X b)
where X is in M[n,m+1], b in R^(m+1) (I hope not in R^(m-1) as you
wrote), e and y in R^n. First of all, you have an error in your expand,
it should read
e'e=y'y-(X b)'y - y'X b + (X b)'X b = y'y - b'X'y - y'X b + b'X'X b.
Now since b'X'y is a real, it's equal to its transpose, so you can add
the two middle terms and get
e'e = y'y - 2 y'X b + b'X'X b.
Next we derivate the function f : R^n --> R with respect to b. Again the
first term is constant so the derivative is zero, the second term is
linear, so it's is own derivative, and the third term is quadratic, so
by the rule for bilinear functions we have
Df(b)(u) = -2 y'X u + u'X'X b + b'X'X u
As before we can apply transpose to the second term and have
Df(b)(u) = -2 y'X u + 2 b'X'X u = 2 ( b'X'X - y'X ) u
So we have the derivative of f at a point b calculated as
Df(b) = 2 ( b'X'X - y'X )
The condition for minimality gives
b'X'X = y'X
or taking transpose on both sides
X'X b = X'y.
So lets come back to Mathematica. First of all, I think you should
consult Statistics`LinearRegression`. To check the validity of the
above formula for a general example now is much easier, since we have a
lot of vectors instead of matrices, so
In[1]:=
y={y1,y1};
b={b1,b2,b3};
X= {{x11,x12,x13},{x21,x22,x23}};
In[2]:= Outer[D,{(y-X.b).(y-X.b)},{b1,b2,b3}][[1]]==2 ( b.Transpose[X].X
- y.X )//
Simplify
Out[2]= True
You notice I didn't use Transpose on vectors, this is due to the
definition of Dot. To allow the natural identification of M(n,1) and so
on with R^n and M(1,1) with R, Dot accepts things like {y1,y2}.{y1,y2},
which in matrix form should be {{y1,y2}}.{{y1},{y2}}. If you insist to
check the above formula with all the transposes, you can do the
following:
In[3]:=
y={{y1},{y1}};
b={{b1},{b2},{b3}};
X= {{x11,x12,x13},{x21,x22,x23}};
In[4]:= Outer[D,Transpose[y-X.b].(y-X.b),{b1,b2,b3}][[1]]==
2 (Transpose[b].Transpose[X].X -Transpose[y].X )//Simplify
Out[4]= True
Yves Gauvreau wrote:
>
>
> I'll give you all of the author solution approch.
>
> """""""""""""""""""
> This is the general linear regression model
>
> Y = XB + E
> eq 1
>
> where
>
> Y is n x 1 matrix of observed values
> X is an n x (m+1) matrix of independent variables
> B is an (m-1) x 1 matrix of the unknown parameters
> E is an n x 1 matrix of error terms
>
> The sums of squares to be minimized is written in matrix form as
>
> E'E = (Y - XB)' (Y - XB) = Y'Y - B'X'Y + B'X'XB eq 2
>
> To minimize this function, we take the derivative with respect to the matrix
> B and get the following:
>
> D[E'E,B] = -2 X'Y + 2X'XB
> eq 3
>
> Equating to zero yields:
>
> (X'X)Bhat = X'Y
> eq 4
>
> The solutions to this matrix equation are
>
> Bhat = inverse(X'X)X'Y
> eq 5
>
> """"""""""""""""""""""
>
> As you can see the final solution is correct and as you mentionned about
> the -2 in equation 3 I had the same concern and that is the reason of my
> query. I wanted to know if I could solve this in Mathematica and arrive at
> the same result. Due to the way M does thing I found that I could not setup
> the problem litteraly and I hoped someone had an idea on how to do it.
>
> Thanks
>
> Yves