Cai T Tony, Ke Zheng T, Turner Paxton
Department of Statistics and Data Science, University of Pennsylvania, Philadelphia, PA, USA.
Department of Statistics, Harvard University, Cambridge, MA, USA.
J R Stat Soc Series B Stat Methodol. 2024 Feb 28;86(4):922-942. doi: 10.1093/jrsssb/qkae003. eCollection 2024 Sep.
Motivated by applications in text mining and discrete distribution inference, we test for equality of probability mass functions of groups of high-dimensional multinomial distributions. Special cases of this problem include global testing for topic models, two-sample testing in authorship attribution, and closeness testing for discrete distributions. A test statistic, which is shown to have an asymptotic standard normal distribution under the null hypothesis, is proposed. This parameter-free limiting null distribution holds true without requiring identical multinomial parameters within each group or equal group sizes. The optimal detection boundary for this testing problem is established, and the proposed test is shown to achieve this optimal detection boundary across the entire parameter space of interest. The proposed method is demonstrated in simulation studies and applied to analyse two real-world datasets to examine, respectively, variation among customer reviews of Amazon movies and the diversity of statistical paper abstracts.
受文本挖掘和离散分布推断应用的启发，我们对高维多项分布组的概率质量函数的相等性进行检验。该问题的特殊情况包括主题模型的全局检验、作者归属的双样本检验以及离散分布的接近度检验。我们提出了一个检验统计量，在原假设下它被证明具有渐近标准正态分布。这种无参数的极限原分布成立，无需每组内的多项参数相同或组大小相等。建立了此检验问题的最优检测边界，并且所提出的检验在整个感兴趣的参数空间内都能达到这个最优检测边界。所提出的方法在模拟研究中得到了验证，并应用于分析两个真实世界的数据集，分别用于检验亚马逊电影客户评论之间的差异以及统计论文摘要的多样性。
