Spectral Methods for Unsupervised and Discriminative Learning with Latent Variables.
Abstract
Incorporating latent or hidden variables is a crucial aspect of statistical modeling. I will present a statistical and a computational framework for guaranteed learning of a wide range of latent variable models under unsupervised and discriminative settings.
It is based on the method of moments, and involves efficient methods for spectral decomposition of low order observed moments (typically up to fourth order). Unsupervised learning of latent variable models such as topic models, hidden Markov models, Gaussian mixtures and network community models is challenging and maximum likelihood estimation of these models is NP-hard. In contrast, we prove that consistent estimation is possible using efficient spectral methods for decomposition of the moment tensors. We establish that the tensor method has low computational and sample complexities. In practice, these methods are fast to implement and embarrassingly parallel, and are thus, scalable to large scale datasets.
Recently, we have provided novel spectral-based approaches for learning discriminative latent variable models such as multi-layer feedforward neural networks and mixtures of classifiers. The moment tensors we construct are based on the label and higher order score functions of the input, in contrast to moments based on the raw input.
The score functions characterize local variation of the probability density function of the input. Thus, incorporating features based on the generative input model is the key ingredient for learning discriminative latent variable models.
Background
Anima Anandkumar is a faculty at the EECS Dept. at U.C.Irvine since August 2010. Her research interests are in the area of large-scale machine learning and high-dimensional statistics. She received her B.Tech in Electrical Engineering from IIT Madras in 2004 and her PhD from Cornell University in 2009. She has been a visiting faculty at Microsoft Research New England in 2012 and a postdoctoral researcher at MIT between 2009-2010. She is the recipient of the Alfred.P. Sloan Fellowship, Microsoft Faculty Fellowship, ARO Young Investigator Award, NSF CAREER Award, IBM Fran Allen PhD fellowship, thesis award from ACM SIGMETRICS society, and paper awards from the ACM SIGMETRICS and IEEE Signal Processing societies.
Research Interests
Her research focus is in the area of inference and learning of probabilistic graphical models and latent variable models. Broadly she is interested in machine learning, high-dimensional statistics, tensor methods, statistical physics, information theory and signal processing.