The Methodology of Knockoffs - Modeling

Two modeling paradigms

In the statement of the variable selection problem, we assume that the response variables are obtained independently from some distribution \(F_{Y|X}\), conditional on the explanatory variables:

\[ \begin{align*} & Y_i | (X_{i,1},\ldots,X_{i,p}) \overset{\text{ind.}}{\sim} F_{Y|X} , & i=1,\ldots,n. \end{align*} \]

This setting is extremely general, since at this point we have specified neither the form of the conditional distribution \(F_{Y|X}\), nor the origin of the explanatory variables \(X\). This broad family of models includes two paradigms in which the knockoff methodology can applied to perform variable selection with FDR control:

• the Model-X paradigm, where the predictors are random;

• the Fixed-X paradigm, where the predictors are fixed.

Under each paradigm, the knockoff methodology has its own strenghts and limitations, as summarized below. Which paradigm should be considered for a given applied problem heavily depends on the data and prior knowledge available to the statistician.

The Model-X paradigm

In the Model-X paradigm, we assume that the explanatory variables are drawn i.i.d. from some distribution \(F_X\). Then, one can write

\[ \begin{align*} & (X_{i,1},\ldots,X_{i,p},Y_i) \overset{\text{i.i.d.}}{\sim} F_{XY} , & i=1,\ldots,n, \end{align*} \]

for some joint distribution \(F_{XY} = F_{Y|X} \circ F_X\) over all variables. In this case, for the knockoff methodology to be applicable,

• the covariate distribution \(F_X\) needs to be known,

• the conditional distribution \(F_{Y|X}\) can be completely arbitrary and unknown,

• the problem can be high-dimensional: no restrictions are imposed on \(n\) and \(p\).

• very powerful and sophisticated machine learning tools can be used to compute \(Z_j\) and \(\tilde{Z}_j\), which only need to satisfy the fairness requirement.

The Fixed-X paradigm

In the Fixed-X paradigm, no assumptions are made on the origin of the explanatory variables (which may be chosen adverserially), but some rather restrictive constraints are imposed on \(F_{Y|X}\). In this case, for the knockoff methodology to be applicable,

• the covariates can be arbitrary and may be considered as fixed,

• the conditional distribution \(F_{Y|X}\) must describe a homoscedastic linear model,

• the problem must be low-dimensional, that is \(n \geq p\).

• the statistics \(Z_j\) and \(\tilde{Z}_j\) must satisfy the additional sufficiency requirement that \(W_j\) only depend on the Gram matrix and on feature-response inner products. That is, we must be able to write

\[ \begin{align*} W_j = W_j\left( [\mathbf{X} \; \tilde{\mathbf{X}}]’ [\mathbf{X} \; \tilde{\mathbf{X}}], [\mathbf{X} \; \tilde{\mathbf{X}}]’ \mathbf{Y} \right). \end{align*} \]

For example, it can be shown that \(Z_j = \hat{\beta}_{j}(\lambda)\), the lasso coefficient with a fixed \(\lambda\), yields a \(W_j\) that satisfies the sufficiency requirement. However, tuning the value of \(\lambda\) by cross-validation would not satisfy the sufficiency requirement.