摘要：Measuring and testing for nonlinear dependence between random vectors is a fundamental problem in statistical inference. Among various measures, the Hilbert-Schmidt independence criterion (HSIC) has garnered significant attention due to its theoretical and computational advantages. We analyze the traditional HSIC in depth and establish a connection between HSIC and the distance correlation. We reveal that it mainly detects linear dependencies and approaches zero when the dimensions grow, with the leading factor being the aggregation of linear dependencies. To improve the sensitivity of HSIC to nonlinear dependencies, we propose the Generalized HSIC (GHSIC), which has a closed form of expression, and equals zero if and only if the two random vectors are independent. Through our investigation, we demonstrate that GHSIC effectively overcomes the limitations of HSIC and exhibits enhanced capability in detecting nonlinear dependencies, particularly in high-dimensional settings. Furthermore, we develop a data-adaptive test based on GHSIC, which outperforms the HSIC-based test in high-dimensional scenarios, even when linear dependencies are present. Extensive numerical experiments demonstrate the superiority of the proposed GHSIC.

 

个人简介：张耀武，上海财经大学信息管理与工程学院教授，数据、算法与工程系系主任，上海数学与交叉学科研究院访问学者。主要从事复杂数据关联分析的统计理论、方法和应用研究，代表性研究成果发表在Annals of Statistics, Biometrika, Journal of Machine Learning Research, Production and Operations Management, Journal of Econometrics, NeurIPS, ICML等顶级期刊和会议上。担任杉数科技科学家顾问，为京东、顺丰、华为、南航等多个国内著名企业提供数据分析和供应链管理等技术服务。