Scalable Empirical Bayes Inference and Bayesian Sensitivity Analysis.

Consider a Bayesian setup in which we observe , whose distribution depends on a parameter , that is, . The parameter is unknown and treated as random, and a prior distribution chosen from some parametric family , is to be placed on it. For the subjective Bayesian there is a single prior in the family which represents his or her beliefs about , but determination of this prior is very often extremely difficult. In the empirical Bayes approach, the latent distribution on is estimated from the data. This is usually done by choosing the value of the hyperparameter that maximizes some criterion. Arguably the most common way of doing this is to let be the marginal likelihood of , that is, , and choose the value of that maximizes . Unfortunately, except for a handful of textbook examples, analytic evaluation of is not feasible. The purpose of this paper is two-fold. First, we review the literature on estimating it and find that the most commonly used procedures are either potentially highly inaccurate or don't scale well with the dimension of , the dimension of , or both. Second, we present a method for estimating , based on Markov chain Monte Carlo, that applies very generally and scales well with dimension. Let be a real-valued function of , and let be the posterior expectation of when the prior is . As a byproduct of our approach, we show how to obtain point estimates and globally-valid confidence bands for the family , . To illustrate the scope of our methodology we provide three detailed examples, having different characters.