Data in various domains, such as neuroimaging and network data analysis, often come in complex forms without possessing a Hilbert structure. The complexity necessitates innovative approaches for effective analysis. We propose a novel measure of heterogeneity, ball impurity, which is designed to work with complex non-Euclidean objects. Our approach extends the notion of impurity to general metric spaces, providing a versatile tool for feature selection and tree models. The ball impurity measure exhibits desirable properties, such as the triangular inequality, and is computationally tractable, enhancing its practicality and usefulness. Extensive experiments on synthetic data and real data from the UK Biobank validate the efficacy of our approach in capturing data heterogeneity. Remarkably, our results compare favorably with state-of-the-art methods in metric spaces, highlighting the potential of ball impurity as a valuable tool for addressing complex data analysis tasks. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Publication:

JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION

http://dx.doi.org/10.1080/01621459.2025.2595733

Author:

Menglu Che

Department of Biostatistics, Yale School of Public Health, New Haven, CT

Ting Li

Department of Statistics & Data Science, Southern University of Science& Technology, Shenzhen, China

Wenliang Pan

State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, China

Xueqin Wang

International Institute of Finance, University of Science and Technology of China, Hefei, China

Heping Zhang

Department of Biostatistics, YaleSchool of Public Health, New Haven, CT