Lightweight Deep Learning on Information Theory

Research topics Deep Learning | Lightweight | Information Theory

Description

Deep learning technique has shown state-of-the-art performance in many fields such as computer vision, natural language processing and many more. However, due to the increasing demand for higher benchmark and rising complexity of the machine learning task, the neural network model is becoming deeper and heavier, which makes it difficult to implement deep learning techniques in battery-powered, memory limited and computation capacity restricted devices. Fortunately, the latest study, including lightweight architecture, neural network quantization, and network pruning, has demonstrated the possibility and potential of lightweight deep learning. But most of the work is based on arbitrary assumptions and lack explanation.

This thesis reviews the study of lightweight deep learning and combines information theory with lightweight deep learning in a novel way. In particular, we use the information bottleneck theory and information plane to analyse pruned neural networks. The preliminary results show that the information plain plotted by MINE estimator reveals the information loss in the pruned neural network and this lays the foundation for the next step information theory-based lightweight deep learning algorithm design.

Output

View Principal Investigator's Publications