Haimonti Dutta

Haimonti Dutta is an Associate Professor in the Department of Management Science and Systems at the State University of New York at Buffalo, NY. She is also a core faculty member of the Computational and Data Enabled Science and Engineering Program at UB. Prior to her current appointment, she served as an Assistant Professor at UB and an Associate Research Scientist at the Center for Computational Learning Systems at Columbia University, NY where she headed the Scalable Analytics Research Group.

Her research broadly focuses on machine learning, distributed optimization, large scale distributed and parallel mining, and probabilistic inference. The federal government and several industry partners have generously supported her research including the National Science Foundation, National Endowment of Humanities, American Institute of India Studies, Amazon Web Services, EMC, Mathworks Inc, Epilepsy Research Foundation and the Consolidated Edison Company of New York.

LinkedIn Profile

Multimodal Deep Learning With An Application To Folk Art Recommendation

In the real world, information can be recorded and stored in multiple modalities. Images often have titles and tags associated with them, songs have lyrics and music to go with them, and videos contain audio-visual signals and inputs from visual, auditory and haptic pathways. Each modality has different statistical properties and useful representations can be learned by combining them and developing joint representations that better represent the data. Deep generative models – models in which distributions can be parameterized by deep neural networks – are often used to model multimodal data. They have been shown to make good inferences by considering heterogeneity and statistical properties of the data.

In this talk, I will first discuss popular deep generative models such as Restricted and Deep Boltzman Machines, Auto-encoders and Variational Auto-Encoders that are extensively used for multimodal learning. Next, I will present recent findings on multimodal learning in an art conservation project involving performing artists who paint scrolls and sing about them.