Mathematics Colloquia and Seminars

Return to Colloquia & Seminar listing

Data balancing across multiple modalities and sources occurs in various forms in many foundation models, including OpenAI’s CLIP and Meta’s DINO. The latter models yield versatile feature representations across numerous domains. We show that data balancing enjoys an unsuspected benefit: reducing the variance of estimators defined as functionals of the empirical distribution over these sources. After describing data balancing as alternating information projections, we will present non-asymptotic statistical bounds quantifying this variance reduction effect. We will discuss how the amount of variance reduction of data balancing can be characterized by eigen-decays of appropriately defined Markov operators and compare it to natural baselines.  We will end with illustrations in contrastive multimodal learning and self-supervised clustering.

This is based on joint work with Lang Liu, Ronak Mehta, Soumik Pal. Preprint <https://arxiv.org/abs/2408.15065>.