Institute of Communications and Connected Systems


Lightweight Deep Learning on Information Theory

Image of a Neuro_Processing_device_ Neuro_Proc.

28 November 2019

Using information theory as a tool to analysis and guide lightweight deep learning architecture design

Funder CSC
Amount £ 48 600


Research topics Deep Learning | Lightweight | Information Theory


Deep learning technique has shown state-of-the-art performance in many fields such as computer vision, natural language processing and many more. However, due to the increasing demand for higher benchmark and rising complexity of the machine learning task, the neural network model is becoming deeper and heavier, which makes it difficult to implement deep learning techniques in battery-powered, memory limited and computation capacity restricted devices. Fortunately, the latest study, including lightweight architecture, neural network quantization, and network pruning, has demonstrated the possibility and potential of lightweight deep learning. But most of the work is based on arbitrary assumptions and lack explanation.

This thesis reviews the study of lightweight deep learning and combines information theory with lightweight deep learning in a novel way. In particular, we use the information bottleneck theory and information plane to analyse pruned neural networks. The preliminary results show that the information plain plotted by MINE estimator reveals the information loss in the pruned neural network and this lays the foundation for the next step information theory-based lightweight deep learning algorithm design.


View Principal Investigator's Publications