Information-Theoretic Approaches to Neural Network Compression, Clustering and Concept Learning
Tuesday, January 09, 2018|
4:00pm - 5:00pm
Forum Hall, Palmer Commons
Add to Google Calendar
About the Event
Deep neural networks have shown incredible performance for inference tasks in a variety of domains, but unfortunately are often enormous cloud-based structures, require significant training data, and are hard for people to interpret. To work towards addressing these challenges, we present three lines of information theoretic investigation. We discuss optimal quantization of synaptic weights and universal lossless compressed representations of feedforward neural networks, taking inferential purpose into account. The basic insight yielding considerable rate reduction is a kind of permutation invariance to the labeling of nodes. We also discuss optimal energy allocation in specialized hardware implementations of neural networks. In considering unsupervised learning, we present asymptotically-optimal algorithms for universal clustering and joint registration/clustering that involve optimization of novel empirical multivariate information measures. Finally we present algorithms for hierarchical, human-interpretable concept learning that iteratively optimize empirical information measures. Such concept learning may act in support of computational creativity.
Lav Varshney is an assistant professor of electrical and computer engineering, computer science, and neuroscience at the University of Illinois at Urbana-Champaign. He is also leading curriculum initiatives for the new B.S. degree in Innovation, Leadership, and Engineering Entrepreneurship in the College of Engineering.
Open to: Public