Conditional inference I

One sample problem (file drawer)

• Basic problem: \(Z \sim N(\mu, 1)\) and we want to report a confidence interval / p-value / estimate only when \(Z > c\).

• Could use two-sided trunction as well… (i.e. Zhong and Prentice)

• Pre-selection our model is \[ {\cal M} = \left\{N(\mu, 1): \mu \in \mathbb{R}\right\} \]

• Selection leads to model \[ {\cal M}^* = \left\{F^*: \frac{dF^*}{dF}(z) \propto 1_{[c,\infty)}(z), F \in {\cal M} \right\} \]

• In this case, the (negative) log-likelihood for observation \(Z\) is \[ \mu \overset{\ell^*}{\mapsto } \frac{1}{2}(z-\mu)^2 + \log(1 - \Phi(c-\mu)) \]

Conditional inference I

One sample problem (file drawer)

• The model \({\cal M}^*\) is an exponential family with sufficient statistic \(T(z)=z\), natural parameter \(\mu\), reference measure \(m_0\) with density \[ z \mapsto \phi(z) 1_{[c,\infty)}(z) \]

• It has cumulant generating function (CGF) \[ \Lambda(\mu) = \frac{\mu^2}{2} + \log(1 - \Phi(c-\mu)) \]

• That is \[ e^{\Lambda(\mu)} = \int_{\mathbb{R}} e^{\mu z} \; m_0(dz) = e^{\mu^2/2} (1 - \Phi(c-\mu)) \]

Conditional inference I

MLE

• Solves \[ Z = \hat{\mu} + \frac{\phi(c-\hat{\mu})}{1 - \Phi(c-\hat{\mu})} \]

Information

• Given by \[ I(\mu) = 1 - \left( \frac{-(c-\mu) \phi(c-\mu)}{1 - \Phi(c-\mu)} + \left(\frac{\phi(c-\mu)}{1 - \Phi(c-\mu)}\right)^2 \right) \]

• Straightforward exercise to show that \[ \lim_{\mu \to -\infty} I(\mu) = 0 \]

Exercise

In lecture, we discussed whether this selection problem can be resolved through a Bayesian approach. Take a flat prior, say \(\mu \sim N(0, 100)\) and write code that constructs 95% posterior intervals (say, equal tailed) with the likelihood in the model \({\cal M}\). When \(\mu=0\), what is the frequentist coverage of such intervals? Using the same prior, repeat with the likelihood in the model \({\cal M}^*\). What is the frequentist coverage at \(\mu=0\)?

Conditional inference I

def CGF(mu, threshold=2):
    c_ = threshold-mu
    return mu**2 / 2 + np.log(normal_dbn.sf(c_))

def grad_CGF(mu, threshold=2):
    # deriv of 1-\Phi(z) is -\phi(z) 
    c_ = threshold-mu 
    return mu + normal_dbn.pdf(c_) / normal_dbn.sf(c_)

def hess_CGF(mu, threshold=2):
    # deriv of -\phi(z) is z \phi(z) 
    c_ = threshold-mu 
    return 1 - (- c_ * normal_dbn.pdf(c_) / normal_dbn.sf(c_) + 
               (normal_dbn.pdf(c_) / normal_dbn.sf(c_))**2)

def MLE(Z, threshold=2):
    
    neg_loglike = lambda mu: (CGF(mu, threshold=threshold) - mu * Z)
    grad_neg_loglike = lambda mu: (grad_CGF(mu, threshold=threshold) - Z)
    mle = fmin_cg(neg_loglike, Z, grad_neg_loglike, disp=False, gtol=1.e-7)[0]
    return mle, grad_neg_loglike(mle), hess_CGF(mle)

Conditional inference I

MLE map

Conditional inference I

Behavior of MLE \(\mu=0, c=2\)

def simulate(truth=0,
             threshold=2,
             ndraw=1000):
    Z = []
    M = []
    I = []
    
    while True:
        z = truth + np.random.standard_normal()
        if z > threshold:
            m = MLE(z, threshold=threshold)
            M.append(m[0])
            Z.append(z)
            I.append(m[2]) # information
        if len(M) > ndraw:
            break
            
    df = pd.DataFrame({'M':M,
                       'Z':Z,
                       'I':I})
    failed = (~np.isfinite(df['M'])).mean()

    df.insert(3, 'wald', (df['M'] - truth) * np.sqrt(df['I']))
    info = hess_CGF(truth)
    df.insert(4, 'theory', (df['M'] - truth) * np.sqrt(info))

    return df.loc[lambda df: np.isfinite(df['M']) * np.isfinite(df['I'])], failed

Conditional inference I

Behavior of MLE \(\mu=0, c=2\)

truth, threshold = 0, 2
norandom, failed = simulate(truth=truth,
                            threshold=threshold)

## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/lib/python3.7/site-packages/scipy/optimize/linesearch.py:156: RuntimeWarning: invalid value encountered in true_divide
##   alpha1 = min(1.0, 1.01*2*(phi0 - old_phi0)/derphi0)
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: invalid value encountered in add
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/lib/python3.7/site-packages/scipy/optimize/linesearch.py:392: RuntimeWarning: invalid value encountered in true_divide
##   alpha1 = min(1.0, 1.01*2*(phi0 - old_phi0)/derphi0)
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: invalid value encountered in add
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/lib/python3.7/site-packages/scipy/optimize/optimize.py:1325: RuntimeWarning: invalid value encountered in multiply
##   pkp1 = -gfkp1 + beta_k * pk
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:3: RuntimeWarning: divide by zero encountered in log
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: divide by zero encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in true_divide
## /Users/jonathantaylor/opt/anaconda3/bin/python3:4: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/bin/python3:5: RuntimeWarning: invalid value encountered in double_scalars
## /Users/jonathantaylor/opt/anaconda3/lib/python3.7/site-packages/pandas/core/computation/expressions.py:194: UserWarning: evaluating in Python space because the '*' operator is not supported by numexpr for the bool dtype, use '&' instead
##   op=op_str, alt_op=unsupported[op_str]

Conditional inference I

print('proportion failures: %f' % failed)

## proportion failures: 0.000999

sns.distplot(norandom['M'])
ax = plt.gca()
ylim = ax.get_ylim()
ax.plot([truth, truth], [0, 20], 'k--')
ax.set_ylim(ylim);
ax.figure

Conditional inference I

Wald pivot

• Usual MLE theory would say that \[ (\hat{\mu} - \mu) \cdot I(\mu)^{1/2} \overset{F_{\mu}^*}{\sim} N(0,1)? \]

• As \(\mu\) is known, it is common to use observed information to form a Wald-type pivot \[ (\hat{\mu} - \mu) \cdot I(\hat{\mu})^{1/2} \overset{F_{\mu}^*}{\sim} N(0,1)? \]

sns.distplot(norandom['theory'], label='MLE theory')
ax = plt.gca()
sns.distplot(norandom['wald'], label='Wald', ax=ax);
Zgrid = np.linspace(-4, 4, 1001)
ax.plot(Zgrid, normal_dbn.pdf(Zgrid), 'k', label='N(0,1)')
ax.legend(fontsize=20)
ax.figure

Conditional inference I

What fails here?

• Standard MLE theory typically has a condition along the lines of \[ n^{-1} I_n(\hat{\mu}_n) \overset{n \to \infty}{\to} I(\mu) > 0 \] where RHS is constant. This certainly is not true here.

• We don’t really have an asymptotic parameter (this is after taking a limit)…

• Observed information can be quite small…

sns.distplot(norandom['I'][np.isfinite(norandom['I'])])

Conditional inference I

Pivotal quantity \(\mu=0, c=2\)

• Let \(F_{\mu} = N(\mu,I) \in {\cal M}\) with corresponding \(F^*_{\mu} \in {\cal M}^*\).

• CDF transform yields \[ P = \frac{F_{\mu}([Z,\infty))}{F_{\mu}([c,\infty))} \overset{F_{\mu^*}}{\sim} \text{Unif}(0,1) \]

pivot = normal_dbn.sf(norandom['Z']) / normal_dbn.sf(threshold)

Conditional inference I

Inference with exact pivot

• Even though standard MLE theory doesn’t hold, you can invert these pivots to construct confidence intervals.

• Low information should translate to wide intervals.

• Indeed Kivaranovic and Leeb show that these intervals have infinite expected length.

def CI(z, threshold=2, level=0.9):

    alpha = (1 - level) / 2
    pivot = lambda mu: normal_dbn.sf(z - mu) / normal_dbn.sf(threshold - mu)
    try:
        upper_lim = bisect(lambda mu: pivot(mu) - (1 - alpha), -20, 20)
        lower_lim = bisect(lambda mu: pivot(mu) - alpha, -20, 20)
        return lower_lim, upper_lim
    except:
        return -np.inf, np.inf
    

CIs = np.array([CI(z) for z in norandom['Z']])
lens = CIs[:,1] - CIs[:,0]
print('bisection failed proportion: %f' % np.mean(~np.isfinite(lens)))

## bisection failed proportion: 0.243129

sns.distplot(lens[np.isfinite(lens)])

Conditional inference I

Some workarounds?

• Could use two-sided conditioning, but some settings you really only want positive looking findings (e.g. drugs with positive effect on a disease).

• Imagining \(Z=n^{1/2}\bar{Z}_n\) (i.e. as arising as a sample mean from a sample of size \(n\)), you could always do data splitting.

Conditional inference I

Data splitting

• Supposing we use \(n/2\) points to select the file drawer and the remaining \(n/2\) for inference, we can write \[ Z = \frac{1}{2}(Z_1 + Z_2) \] where \[ (Z_1, Z_2) \sim N(\mu (1, 1), 2 \cdot I_2) \]

• Define \[ \Delta = \frac{1}{2}(Z_1 - Z_2) \sim N(0, 1) \] so that \[ \text{Cov}(Z, \Delta)=0. \]

• Note that \[ Z_1 = Z + \Delta \]

so that \[ \left\{Z_1 > c\right\} = \left\{Z + \Delta > c\right\} \]

• Finally, note that we can always add \(\Delta\) to \(Z\) to form an effective \(Z_1\) even if \(Z\) did not arise from a sample mean.

• We can of course change the variance of \(\Delta\) by setting \(\Delta \sim N(0, \tau^2)\).

Exercise

• When data splitting we must choose some proportion of the data. Suppose we use \(n_1 = \pi n\) of the data for splitting. What is the corresponding value of \(\tau\)?

Conditional inference I

Data splitting

• Usually data splitting returns just \(\hat{\mu}_{\text{split}}(Z)=Z_2\) but we can treat it completely analogously to file drawer problem with no randomization.

• Pre-selection, let’s take our model to be the laws of \((Z,\Delta)\) \[ {\cal M} = \left\{N\left( \begin{pmatrix} \mu \\ 0 \end{pmatrix}, \begin{pmatrix} 1 & 0 \\ 0 & \tau^2 \end{pmatrix} \right): \mu \in \mathbb{R}\right\} \] which we truncate to \[ A = \left\{(z,\delta) :z+\delta > c\right\} \] yielding \[ {\cal M}^* = \left\{F^*: \frac{dF^*}{dF}(z,\delta) \propto 1_A(z,\delta) \right\} \]

• This is an exponential family with sufficient statistic \(T(z,\delta)=z\), natural parameter \(\mu\) and reference measure \(m_0\) having density \[ (z,\delta) \to \phi_{0,\Sigma}(z,\delta) 1_A(z,\delta) \]

• Its CGF is \[ \mu^2 / 2 + \log\left(1 - \Phi((c - \mu) / \sqrt{1 + \tau^2})\right) \]

Conditional inference I

MLE of data splitting

• Solves \[Z = \hat{\mu} + \frac{1}{\sqrt{1 + \tau^2}} \cdot \frac{\phi((c-\hat{\mu}) / \sqrt{1+\tau^2})}{1 - \Phi((c-\hat{\mu}) / \sqrt{1 + \tau^2})} \]

• Earlier we simply had \(\tau=0\).

noise_sd = 1

def CGF(mu, threshold=2, noise_sd=0):
    c_ = (threshold-mu) / np.sqrt(1 + noise_sd**2)
    return mu**2 / 2 + np.log(normal_dbn.sf(c_))

def grad_CGF(mu, threshold=2, noise_sd=0):
    # deriv of 1-\Phi(z) is -\phi(z) 
    c_ = (threshold-mu) / np.sqrt(1 + noise_sd**2)
    scaling = -1 / np.sqrt(1 + noise_sd**2) 
    return mu + scaling * (-normal_dbn.pdf(c_) / normal_dbn.sf(c_))

def hess_CGF(mu, threshold=2, noise_sd=0):
    # deriv of -\phi(z) is z \phi(z) 
    c_ = (threshold-mu) / np.sqrt(1 + noise_sd**2)
    scaling = 1 / (1 + noise_sd**2)
    return 1 - scaling * (- c_ * normal_dbn.pdf(c_) / normal_dbn.sf(c_) + 
                           (normal_dbn.pdf(c_) / normal_dbn.sf(c_))**2)

def MLE(Z, threshold=2, noise_sd=0):
    
    neg_loglike = lambda mu: (CGF(mu, threshold=threshold, noise_sd=noise_sd) - mu * Z)
    grad_neg_loglike = lambda mu: (grad_CGF(mu, threshold=threshold, noise_sd=noise_sd) - Z)
    
    mle = fmin_cg(neg_loglike, Z, grad_neg_loglike, disp=False, gtol=1.e-7)[0]
    return mle, grad_neg_loglike(mle), hess_CGF(mle, noise_sd=noise_sd)

Conditional inference I

MLE map with \(\tau > 0\)

Conditional inference I

Comparing information

Conditional inference I

Behavior of MLE \(\mu=0, \tau=1, c=2\)

def simulate_random(truth=0,
                    threshold=2,
                    noise_sd=1,
                    ndraw=1000):
    Z = []
    M = []
    I = []
    
    while True:
        z = truth + np.random.standard_normal()
        if z + noise_sd * np.random.standard_normal() > threshold:
            m = MLE(z, threshold=threshold, noise_sd=noise_sd)
            M.append(m[0])
            Z.append(z)
            I.append(m[2]) # information
        if len(M) > ndraw:
            break
            
    df = pd.DataFrame({'M':M,
                       'Z':Z,
                       'I':I})
    df.insert(3, 'wald', (df['M'] - truth) * np.sqrt(df['I']))
    info = hess_CGF(truth, noise_sd=noise_sd)
    df.insert(4, 'theory', (df['M'] - truth) * np.sqrt(info))

    return df

Conditional inference I

Behavior of MLE \(\mu=0, c=2\)

Conditional inference I

Behavior of Wald pivot \(\mu=0, \tau=1, c=2\)

Conditional inference I

Behavior of MLE \(\mu=-1, c=2\)

Observed information

Conditional inference I

Behavior of MLE \(\mu=-1, \tau=1, c=2\)

Sampling distribution of \(\hat{\mu}\)

Conditional inference I

Behavior of Wald pivot \(\mu=-1, \tau=1, c=2\)

Conditional inference I

Back to data splitting

• We motivated this estimator on the basis of data splitting though this estimator is not simply the data splitting estimator \(Z_2 = Z - \Delta \sim N(\mu, 2)\) (when \(\tau=1\)).

• When \(\tau \neq 1\) we can take \[ Z_2 = Z - \frac{1}{\tau^2} \Delta \sim N\left(\mu, 1 + \frac{1}{\tau^2}\right) \] Note that \(\text{Cov}(Z_2, Z_1)=0\).

Conditional inference I

Back to data splitting

• When \((Z_1, Z_2)\) are both observed we can condition on \(Z_1\) – this yields data splitting.

• It is perhaps easier to transform the original model to work with \((Z_1, Z_2)\) instead of \((Z, \Delta)\). This model is \[ \bar{\cal M} = \left\{N\left( \begin{pmatrix} \mu \\ \mu \end{pmatrix}, \begin{pmatrix} 1 + \tau^2 & 0 \\ 0 & 1 + \frac{1}{ \tau^2} \end{pmatrix} \right): \mu \in \mathbb{R}\right\} \] which we restrict to \[ \bar{A} = \left\{(z_1, z_2): z_1 > c \right\} \] yielding model \(\bar{\cal M}^*\).

• Its restriction to \(\bar{A}\) is again an exponential family (with identical CGF) and natural parameter and the sufficient statistic is \[ \bar{T}(z_1,z_2) = \frac{Z_1 + \tau^2 Z_2}{1 + \tau^2}. \]

• Further conditioning on \(Z_1\) yields a finer set of conditional laws \[ \bar{\cal M}^*(z_1) \equiv \left\{N\left(\mu, 1 + \frac{1}{\tau^2}\right) \right\} \] i.e. the collection \[ {\cal L}(Z_2 |Z_1=z_1, z_1>c) \overset{F_{\mu}}{=} N\left(\mu, 1 + \frac{1}{\tau^2}\right). \]

Conditional inference I

Back to data splitting

• As \(\mu \to -\infty\) then under \(F_{\mu^*} \in {\cal M}^*\) it is not hard to see that \(Z_1 \overset{P}{\to} c\) so \[ Z \overset{P}{\to} \frac{c + \tau^2 Z_2}{1 + \tau^2} \] and we might expect the MLE to converge to \(\frac{(1 + \tau^2) Z - c}{\tau^2} \approx Z_2\) under \(F_{\mu}^*\).

• As \(\mu \to \infty\), \(F_{\mu}^*(A) \to 1\) and \(\hat{\mu}(Z) \to Z\).

• This MLE is adaptive in that it can achieve the efficient estimator \(Z\) of \({\cal M}\) when selection is minimal but reverts to \(Z_2\) the data splitting estimator under rarer selection events.

Conditional inference I

MLE map

Conditional inference I

Confidence intervals

• In the noiseless case, we had an exact pivot based on the truncated normal distribution based on CDF transform under \(H_0:\mu=\mu_0\).

• In data splitting case, similar construction will work from general one-dimensional exponential family considerations.

• For a natural parameter value \(\mu\) we use as reference the law of sufficient statistic \(T(z,\delta)=z\) under \(F_{\mu}^*\).

• Under \(F_{\mu}^*\), \(Z\) has density proportional to \[ \begin{aligned} z &\mapsto e^{\mu z} \cdot \int_{\mathbb{R}} \phi_{0, \Sigma}(z,\delta) 1_A(z,\delta) \; d\delta \\ & \propto e^{\mu z} \cdot \phi(z) \cdot (1 - \Phi((c - z) / \tau)) \end{aligned} \]

• Not hard to squint at this to see that as \(\tau \downarrow 0\) \[ (1 - \Phi((c - z) / \tau)) \to 1_{[c,\infty)}(z) \] so this is a smoothed out version of our earlier constraint.

• This distribution no longer has a closed form for \(\tau > 0\) but it is a simple enough 1-dimensional distribution.

Conditional inference I

One-parameter exponential family

• Setting the reference measure to be \[ m = \sum_{i} w_i \cdot \delta_{z_i} \] one can construct the exponential family \[ \left\{P_{\mu}: \frac{dP_{\mu}}{dm}(z) \propto e^{\mu z} \right\} \] with CGF \[ e^{\Lambda(\mu)} = \int_{\mathbb{R}} e^{\mu z} \; m(dz) = \sum_i e^{\mu z_i} w_i. \]

• CDF transform for observed value \(Z\) is simply \[ \int_{(-\infty, Z]} P_{\mu}(dz) = \frac{\sum_{i:z_i \leq Z} e^{\mu z_i} w_i}{\sum_{i} e^{\mu z_i} w_i} \]

• Choice of \((Z, W)\) include equally spaced grid for \(Z\) and \(W= \phi(Z) \cdot (1 - \Phi((c-Z)/\tau))\) or \(Z_b \overset{IID} N(0,1), 1 \leq b \leq B\) and \(W_b = 1 - \Phi((c-Z_b) / \tau)\).

Conditional inference I

One-parameter exponential family

from selectinf.distributions.discrete_family import discrete_family

Z = np.random.standard_normal(200)
W = np.random.sample(Z.shape)
F = discrete_family(Z, W)
Z_obs, mu = 1.2, 0.4
F.cdf(mu, Z_obs), np.sum((Z <= Z_obs) * np.exp(mu * Z) * W) / np.sum(np.exp(mu * Z) * W)

## (0.825927138173137, 0.8259271381731371)

Conditional inference I

One-parameter exponential family

def find_interval(Z, guess_for_grid, noise_sd):
    W = normal_dbn.pdf(guess_for_grid) * normal_dbn.sf((threshold - guess_for_grid) / noise_sd)
    F = discrete_family(guess_for_grid, W)
    mle = MLE(Z, threshold=threshold, noise_sd=noise_sd)[0]
    l, u = F.equal_tailed_interval(Z, alpha=0.05)
    return pd.DataFrame({'Z':[Z],
                         'L':[l],
                         'U':[u],
             'M':[mle]}) 

df = pd.concat([find_interval(z, np.linspace(z-6,z+6,501), noise_sd)
                for z in np.linspace(-4,8,301)])

Conditional inference I

One-parameter exponential family

## (-14.03577263244398, 11.115152436823685)

Confidence intervals are adaptive: well above threshold they are unadjusted; well below they are the data splitting intervals.

Conditional inference I

One-parameter exponential family

## (0, 6)