MA 575 – Fall 2022. Final Exam
Some useful formulas
• The Gaussian distribution N(µ, σ2), µ 2 R, σ2 > 0 has pdf x 2 R. If X ~ N(µ, σ2), we have E(X) = µ, Var(X) = σ2.
• Throughout the exam we consider a multiple linear regression model
The parameters of the model are β 2 Rp and σ2 > 0 with true values β?, σ?
2 respectively. Throughout we assume that the model includes an intercept.
• (Woodbury identity) Let A 2 Rm⇥m invertible, and u, v 2 Rm be such that 1 + v'
A−1u ≠ 0. Then A + uv' is invertible, and
• We recall also that if and det(A) = A11A22 − A12A21 ≠ 0, then A is invertible and
Problem 1: Consider the linear regression model given in (1).
a. (1pt) TRUE or FALSE: the model assumes that the components of y are independent with the same distri bution.
b. (1pt) TRUE or FALSE: the model is not applicable when the explanatory variables are not continuous.
c. (1pt) If βˆ denotes the least squares estimate of β in model (1), use y, X, βˆ to express the vector of fitted values yˆ, and its covariance matrix.
d. (2pt) Consider model (1) with p = 2 (simple linear model). Let x = (x1,...,xn)0 denote the unique ex-planatory variable of the model (recall that the model contains an intercept). Let xc = (x1−x, . . . , x ¯ n− ¯x)0
, and 1 = (1,..., 1)' 2 Rn, where x¯ = xi/n. Use a projection argument to show that the vec-tor of fitted values yˆ of the model can be written as
Give the expression of β˜0 and β˜1.
e. (1pt) Consider again the case p = 2, and assume that xi = 0. Find Var(βˆ0), and Var(βˆ1).
Problem 2: Consider the linear regression model given in (1).
a. (2pts) The least squares estimator of β is β that minimizes the function β 7! ky − Xβk2. Give the expression of βˆ and the expression of an unbiased estimator σˆ2 of σ2.
b. (2pts) Under the assumptions of the model what are the distributions of βˆ and σˆ2?
c. (2pts) Suppose that we modify model (1) to y = Xβ + , where ~ N(0, σ2⌦−1) for a symmetric positive define matrix ⌦ 2 Rp⇥p assumed known. In that case we estimate β using β that minimizes the function β 7! (y − Xβ)0
⌦(y − Xβ). Give the expression of βˇ and the expression of an unbiased estimator σˇ2 of σ2 in this model.
Problem 3: Let be a Gaussian random vector with mean and covariance matrix given by
a. (1pts) Answer TRUE or FALSE: the variables Y1, Y2, Y3 as given are iid.
b. (1pts) Answer TRUE or FALSE: the variables Y1, Y3 are independent.
c. (1pts) Give the expression of the probability density function (pdf) of 2Y1.
(d) (1pts) Let Z = Y1
2 + (Y2 − 1)2. Find E(Z).
(d) (2pts) Find the expectation and the covariance matrix of
Problem 4: We consider the linear regression model in (1). Consider a sub-model y = X1β1 + , where X1 2 Rnxp1 is a sub-matrix of X that collects only p1 of the p columns of X. Let X2 2 Rn⇥p2 be the remaining columns of X, with p1 + p2 = p. We partition accordingly the true value β? as (β?
T
,1, β?
T
,2)T. The AIC of the sub-model is
with a similar expression for the full model.
a. (1pts) Answer TRUE or FALSE: in general, adding more explanatory variables to a linear model tend to produce fitted values with high biases, whereas removing many explanatory variables from the model tend to produce fitted values with high variances.
b. (1pts) Answer TRUE or FALSE: In general, when comparing models, the AIC and the R2 typically yield the same conclusion.
c. (1pts) Show that in the set up described at the beginning, if β?,2 = 0, then we have E(ky − X1βˆ1k2) = σ?
2(n − p1).
d. (1pts) By looking at the derivative of the function log(1 − x) + x, show that −x − x2 ≤ log(1 − x) ≤ −x for all x 2 [0, 1/2).
e. (2pts) In the specific set up described at the beginning, use the above to show that when n is larger than p, and β?,2 = 0, the smaller model is typically preferred according to the AIC criterion.
Problem 5: We consider the linear regression model in (1). Let y(i) 2 Rn−1 be the vector of responses obtained after removing the i-th response. Let X(i) 2 R(n−1)⇥p be the explanatory matrix obtained after removing the i-th row of X, that we denote xi. Let βˆ
(i) be the least squares estimate of the model y(i) = X(i)β + (i).
(a) (1pts) The leverage of the i-th observation is hi = xi(X' X)−1x' i. Answer TRUE or FALSE: small value of hi means that the i-th observation is likely an outlier in the x-space.
(b) (1pts) Let the residuals of the model be ˆ. Use the fact that Var(ˆ) = σ?
2(In−H) to show that 0 ≤ hi ≤ 1 for all i.
(c) (2pts) Suppose that p = 2 (simple linear model). Let (x1,...,xn)0 denote the unique explanatory variable of the model (recall that the model contains an intercept). We set x¯ = xi/n. Show that in this case the leverage of the i-th observation can be written as
(d) (2pts) In the general set up above, the i-th studentized residual is defined as where ✏(i) = yi − xi
ˆβ(i) and σˆ(
2
i) = ky(i) − X(i) ˆ β(i)k2/(n − p − 1). Use the relation ˆβ(i) = ˆβ − (X' X)−1x0
i to show that