STAT3006 Assignment 4—High-Dimensional Inference
Due Date: 15th November 2021
Weighting: 25%
Instructions
• The assignment consists of three (3) problems; Problems 1 and 2 are worth 10 Marks
each, and Problem 3 is worth 5 Mark. Each Mark is equally weighted and is worth 1%
of the overall course grade.
• The mathematical elements of the assignment can be completed by hand, in LaTeX (prefer-
ably), or in Word (or other typesetting software). The mathematical derivations and ma-
nipulations should be accompanied by clear explanations in English regarding necessary
information required to interpret the mathematical exposition.
• Computation problems can be answered using your programming language of choice, al-
though R is generally recommended, or Python if you are uncomfortable with R. As with
the mathematical exposition, you may choose to typeset your answers to the problems in
whatever authoring or word processing software that you wish. You should also maintain a
copy of any codes that you have produced.
• Computer generated plots and hand drawn graphs should be included together with the text
where problems are answered.
• The assignment will require four (4) files containing data, that you can can download from
the Assignment 4 section on Blackboard. These files are: zip.txt, golub_genes.csv,
golub_labels.csv, and prostate.csv.
• Submission files should include the following (which ever applies to you):
– Scans of handwritten mathematical exposition.
– Typeset mathematical exposition, outputted as a pdf file.
– Typeset answers to computational problems, outputted as a pdf file.
– Program code/scripts that you wish to submit, outputted as a txt file.
• All submission files should be labeled with your name and student number and
archived together in a zip file and submitted at the TurnItIn link on Blackboard.
We suggest naming using the convention:
FirstName_LastName_STAT3006A4_[Problem_XX/Problem_XX_Part_YY].[FileExtension].
• As per my.uq.edu.au/information-and-services/manage-my-program/student-in
tegrityand-conduct/academic-integrity-and-student-conduct, what you submit
should be your own work. Even where working from sources, you should endeavour to write
in your own words. You should use consistent notation throughout your assignment and
define whatever is required.
Problem 1 [10 Marks]
Consider the data set zip.txt, which contains n = 7291 rows of data, where each row is an
observation
Z>i =
(
Yi,X
>
i
) ∈ R1+q,
with q = 256. Here Yi ∈ {0, 1, . . . , 9} is a label indicating the digit that is represented by the
vectorized 16× 16 matrix (image) Xi ∈ Rq.
Part 1 [1 Mark]
Select one of the digits y ∈ {0, 1, . . . , 9} and plot m = 9 unique images with Yi = y, in the
same plot, as characterized by Xi .
Part 2 [2 Marks]
Using all n = 7291 observations of the sample X˜n =
(
X˜1, . . . , X˜n
)
, where X˜i = Xi−X¯n (i ∈ [n])
and X¯n = n−1
∑n
i=1Xi, obtain a solution to the optimization problem:
Fˆ, Rˆ = arg min
F∈Rs×q ,R∈Rq×s
n∑
i=1
∥∥∥X˜i −RFX˜i∥∥∥2
2
,
for s = 4 and report the minimum value:
n∑
i=1
∥∥∥X˜i − RˆFˆX˜i∥∥∥2
2
.
Part 3 [2 Marks]
Using the results from Part 2, plot the forward mappings Wˆ1, . . . , Wˆn, where Wˆi = FˆX˜i, for
each i ∈ [n], colored by the labels Yi. Discuss whether there appears to be differences in the
distributions of the forward mapped observations Wˆ1, . . . , Wˆn, corresponding to different values
of the labels Yi.
Part 4 [1 Mark]
Via a spectral decomposition of the Grammian
G˜ =
n∑
i=1
X˜iX˜
>
i
report the proportion of total variance that is explained by s = 4 eigenvectors corresponding to
the first s largest eigenvalues.
Part 5 [2 Marks]
Using all n = 7291 observations of the sample Xn = (X1, . . . ,Xn), obtain a solution θˆ =(
σˆ2, µˆ, Rˆ
)
to the optimization problem
arg max
θ
logLn (θ) ,
where
logLn (θ) =
n∑
i=1
log φ
(
Xi;µ,RR
> + σ2Iq
)
= −n
2
q log (2pi)− n
2
log
∣∣RR> + σ2Iq∣∣−−1
2
trace
([
RR> + σ2Iq
]−1
S (µ)
)
,
S (µ) = n−1
∑n
i=1 (Xi − µ) (Xi − µ)>, and θ = (σ2,µ,R), with σ2 > 0, µ ∈ Rq, and R ∈ Rq×s,
for s = 4.
Part 6 [1 Marks]
Using the results from Part 5, estimate the posterior expectations of the latent variables
Wi:
E (Wi|Xi) =
(
R>R+ σ2Is
)−1
R> (Xi − µ)
for each i ∈ [n], and plot the estimated posterior expectations, colored by the labels
Yi.
Part 7 [1 Marks]
Using all n = 7291 observations of the sample Xn = (X1, . . . ,Xn), use 3-layer autoencoder
with some activation function a : R → R (of your choice) to obtain an s-dimensional
nonlinear dimensionality reduction
Wi = F (Xi) , for each i ∈ [n]
and for s = 4, where
F : Rq → Rs, x 7→ (a (f1x+ c1) , . . . , a (fsx+ cs)) ,
for some vectors f1, . . . , fs ∈ Rq and scalars c1, . . . , cs ∈ R. Then, plot W1, . . . ,Wn, colored by
the labels Yi.
Problem 2 [10 Marks]
Consider the data set golub_genes.csv , which contains q = 3571 rows of data corresponding
observations Xi ∈ Rq (i ∈ [n]), in each of the n = 72 columns. Here, each row corresponds to
the expression levels of a gene j across the n cells, corresponding to the columns. The data set
golub_labels.csv then contains the corresponding label corresponding to the cell type of each of
the n columns of golub_genes.csv, where the cells are either labeled as “ALL” or “AML”, where
ALL stands for Acute Lymphoblastic Leukemia, and AML stands for Acute Myeloid Leukemia.
For each i ∈ [n], we will write Yi = 1 if cell i is ALL and Yi = 2 if cell i is AML.
Part 1 [2 Mark]
Let f1 (x) = f (x|Y = 1) and f2 (x) = f (x|Y = 2) be the probability density functions of the
gene expression levels for ALL and AML cells, respectively. Using an maximum mean discrepancy
statistic with kernel of the form.:
κg (x,y) = g
(‖x− y‖22) ,
where g (t) = exp {−βt} for β = 2−28, and using the data X1, . . . ,Xn and Y1, . . . , Yn, (assumed to
be independent and identically distributed), test the hypotheses
H0 : f1 = f2 versus H0 : f1 6= f2,
at the α = 0.1 significance level. That is, report the test statistic, critical value, and decision
that is made. If your decisions is to not reject the null hypothesis H0, then comment on whether
or not you believe that the test was powerful enough to reject the null hypothesis,
based on the sample size.
Part 2 [2 Mark]
Let f1j (x) = f (x1|Y = 1) and f2j (x) = f (x2|Y = 2) be the marginal probability density functions
of the gene expression levels of the jth gene for ALL and AML cells, respectively. Let
P (f) =
∫
R
xf (x) dx
be the mean of univariate probability density function f : R → R≥0. For each j ∈ [q], compute
a p-value Pj for a test of the hypotheses
H0 :P (f1j) =P (f2j) versus H0 :P (f1j) 6=P (f2j) ,
using the data X1, . . . ,Xn and Y1, . . . , Yn. Plot the p-values Pq = (P1, . . . , Pq) using a his-
togram.
Part 3 [2 Mark]
Following from Part 2, plot the empirical cumulative distribution function (ECDF) for
the sample of p-values:
F (p;Pq) =
1
q
q∑
j=1
Jp ≤ PjK ,
along with the cumulative distribution function of the uniform. distribution on the
domain [0, 1]. Comment on whether the distribution of p-values is sub-uniform. or not,
and whether or not this observation conforms with the conclusions made in Part 1.
Part 4 [2 Mark]
Using the Benjamini–Hochberg and Benjamini–Yekutieli methods, identify sets of genes j ∈ [q]
that are significant at the false discovery rate controlled level of αFD = 0.05. Report
how many of the genes are significant under each method, and report the largest
p-value that was rejected under each method.
Part 5 [1 Mark]
Using exploratory techniques and the samples X1, . . . ,Xn and Y1, . . . , Yn, explain whether you
believe that the outcomes from either of the methods applied in Part 4 are valid by
way of a discussion of the required assumptions.
Part 6 [1 Mark]
Prove that ∞∑
j=1
δ (min {j,m})
j (j + 1)
≤ 1,
for δ (k) = (2m)−1 k (k + 1), and provide a false discovery rate step-up rejection proce-
dure based on this observation.
Problem 3 [5 Marks]
Let (X1, Y1) , . . . , (Xn, Yn) be an independent and identically distributed sample of pairs of covari-
ates and responses, where Xi ∈ Rq and Yi ∈ R, where q may be larger than n.
Suppose that
E [Yi|Xi = xi] = β>xi, (1)
for some potentially sparse β ∈ Rq (here, we take sparse to mean that many of the coordinates
βj = 0 for many j ∈ [q], where β> = (β1, . . . , βq). We wish to estimate β via the so-called elastic
net-penalized least squares estimator:
βˆ = arg min
β∈Rq
1
2n
n∑
i=1
(
Yi − β>Xi
)2
+ λ
{‖β‖1 + ‖β‖22} , (2)
for some λ ≥ 0.
Part 1 [1 Mark]
Argue that Problem (2) is equivalent to the problem:
βˆ = arg min
β∈B(γ)
1
2n
n∑
i=1
(
Yi − β>Xi
)2 , (3)
where B (γ) =
{
β ∈ Rq : ‖β‖1 + ‖β‖22 ≤ γ
}
, for some γ > 0.
Part 2 [1 Mark]
Plot the set B (γ) =
{
β ∈ Rq : ‖β‖1 + ‖β‖22 ≤ γ
}
for some value of γ and discuss whether
you believe that elastic net-penalized least squares estimator can be sparse.
Part 3 [2 Mark]
Devise an algorithm for solving either Problem 2 or Problem 3.
Part 4 [1 Mark]
The rows of the file prostate.csv contains n = 98 pairs of covariates Xi ∈ Rq (q = 8; in the first
8 columns) and response Yi (in the 9th column), for i ∈ [n]. Assuming that these data admit the
relationship (2), compute estimates of β for these data using the elastic net-penalized
least squares estimator for some increasing sequence of γ (or decreasing sequence of
λ ≥ 0). Plot the trajectory of the sequence of estimates as γ increases (or λ decreases).