Course Introduction

STATS 305C

Common tasks

Data matrix

• Basic data block: data matrix \(X \in \mathbb{R}^{n \times p}\).

• Usual model: \(X_i=X[i] \sim F\).

The exam score data

One sample problem

• Let \(\mu \in \mathbb{R}^p, \Sigma \in \mathbb{R}^{p \times p}\).

• Do these data come from distribution \(N(\mu, \Sigma)\)?

• Or, \(F \overset{?}{=} N(\mu,\Sigma)\)?

Picking a statistic

• Is \(T\) big?

• Is it the “right” statistic?

Two sample problem

• Two data matrices: \(X_1 \in \mathbb{R}^{n_1 \times p}, X_2 \in \mathbb{R}^{n_2 \times p}\).

• Basic model: \(X_1[i] \sim F_1\), \(X_2[i] \sim F_2\).

• Usual question: \(F_1 \overset{?}{=} F_2\)?

Picking a statistic

• Let’s artificially made a group here: 1:50 in group 1, 51:88 in group 2.

Multi sample problem

• Many data matrices: \(X_k \in \mathbb{R}^{n_k \times p}, 1 \leq k \leq K\).

• Basic model: \(X_k[i] \sim F_k\)

• Usual question: \(F_k \overset{?}{=} F \ \forall k\)?

Regression

• Response matrix: \(Y \in \mathbb{R}^{n \times q}\)

• Design matrix: \(X \in \mathbb{R}^{n \times p}\)

Regression with multivariate response

• Usual task:

\[ \hat{B} = \text{argmin}_B \|Y - XB\|^2_F \]

• What if errors are correlated?

• Penalized form?

Frobenius norm

• Plays role of squared error in multivariate problems

\[ \|A\|_F^2 = \text{Tr}(A^TA) \]

• Corresponds to inner product:

\[ \langle A, B \rangle = \text{Tr}(A^TB) \]

Multiple-response regression

Inference in multiple-response regression

Inference in multiple-response regression

Classification

• Response matrix: \(Y \in \mathbb{R}^{n \times K}\): each row of \(Y\) is 1-hot – indicator of category

• Design matrix: \(X \in \mathbb{R}^{n \times p}\)

Classification beyond binary

• Usual question: for some classification loss

\[ \hat{f} = \text{argmin}_f{\cal L}(Y; f(X)) \]

Generative classifiers

• LDA, QDA, Naive Bayes

Discriminative

• Multinomial logistic + variants

• RKHS methods (inference?)

Linear Discriminant Analysis

Matrix factorization

• Find \(U, D, V\) such that \[ X = U D V^T \]

• PCA the “most popular” exploratory technique… SVD in disguise

SVD

PCA

Proportional asymptotics

• We’ll explore some properties of PCA and related methods as \(n, p\) both grow with \(n/p \to \gamma\)

• (Maybe) some implications for regression and other methods under proportional asymptotics

Matrix approximation

• Find \(\hat{U}, \hat{D}, \hat{V}\) such that

\[ X \approx \hat{U} \hat{D} \hat{V}^T \]

Other methods for matrix approximation

• Low-rank approximation to \(X\) (low-rank SVD)

• Structure in \(\hat{U}, \hat{V}\)? (Non-negative matrix factorization, sparse PCA, …)

Covariance matrix approximation

• Diagonalize

\[ \Sigma = \text{Cov}_F(X) = VDV^T \]

Factor analysis

• Factor analysis models as low-rank + diagonal

\[ \Sigma = \text{Cov}_F(X) = \Gamma \Gamma^T + \text{diag}(\psi) \]

Canonical correlation analysis (CCA)

Find common structure in different feature sets

• Given groups of measurements \(Y, X\) on same individuals (so data matrix is \(\mathbb{R}^{n \times (n_Y + n_X)}\).

CCA

• Example: \(Y\) is a set of phenotypes, \(X\) is a set of genes.

• Want to find functions of \(Y\) related to functions of \(X\):

\[ \text{maximize}_{(a,b): \text{Var}(a^TY)=\text{Var}(b^TX)=1} \text{Cov}(a^TY, b^TX). \]

Common structure in different feature sets

• Given groups of measurements \(X_1, \dots, X_2, \dots, X_K\) on same individuals (so data matrix is \(\mathbb{R}^{n \times (n_1 + \dots + n_K)}\).

• Want to find functions of \(X_k\)’s that are all highly “related” to one another.

Visualization: low-dimensional projection

• Given \(X \in \mathbb{R}^{n \times p}\), find \(\hat{X} \in \mathbb{R}^{n \times k}\) with \(k \ll p\) such that

\[ d(X[i], X[j]) \approx d_{\ell_2}(\hat{X}[i], \hat{X}[j]) \]

Low-dimensional projection

• Examples: principal component scores (biplot), t-SNE, “manifold learning”

Low-dimensional projection using only distance

• Given pairwise “distances” \(D \in \mathbb{R}^{n \times n}\) find \(\hat{X} \in \mathbb{R}^{n \times k}\) with \(k\) small such that

\[ D_{ij} \approx d_{\ell_2}(\hat{X}[i], \hat{X}[j]) \]

Multidimensional scaling

Clustering

• Given \(X \in \mathbb{R}^{n \times p}\), partition \(\{1, \dots, n\}\) into “homogeneous” clusters.

• Often can be done using only pairwise distances \(D \in \mathbb{R}^{n \times n}\).

Hierarchical clustering

Mixture modeling

• Soft version of \(K\)-means

• EM algorithm

Mixed models (time permitting)

\[ Y | X, Z \sim N(X\beta + Z\alpha, \Sigma) \]

• Random effects: \(\alpha \sim N(0, \bar{\Sigma})\)

• EM approach

• Connection to ridge, GAM, other kernels

• Inference?

Generalized linear mixed models (time permitting)

Logistic

\[ P(Y=1 | X, Z) = \frac{e^{X\beta+Z\alpha}}{1 + e^{X\beta+Z\alpha}} \]

• Random effects: \(\alpha \sim N(0, \bar{\Sigma})\)

• Difficulty: likelihood doesn’t have a closed form…

• Inference?