Overview

Deep learning is a subset of machine learning, where artificial neural networks - algorithms inspired by the human brain - learn from large amounts of data. 

The field of natural language processing (NLP) is one of the most useful application areas of deal learning, which is now integrated into various aspects of daily life, from smartphones to search engines. 

In this course you'll explore the intersection of moden deep-learning technologies with the fundamental concepts of NLP. 

You'll master cutting-edge techniques, and gain the skills to move from the basics of deep learning to implementing complex deep models for real-world NLP applications including dialogue systems, automatic summarisation and translation. 

You'll have lab sessions and assignments specifically targeted at putting some of the learned technologies into practical use by implementing and testing them on real-world natural text datasets. 

Content and structure

The short course will be taught through lectures and laboratory sessions.

You'll cover: 

Foundations of Deep Learning and NLP:

• Backpropagation and Automatic Differentiation
• Log-linear modeling
• NLP Feature Engineering
• Softmax, MLE estimation, Probabilistic models of NLP
• Text classification
• Feed-forward NNs, Sentiment Analysis 

NLP Representation Learning

• Word and Sentence Representations (Encoding Words, Pooling and n-grams, Word Embeddings, Unsupervised Word Repr., Skip-Gram, BERT)
• Contextualized Representation

Structured Prediction and Language Modeling

• Neural n-gram models
• Recurrent Neural Networks (Elman networks, GRUs, LSTMs, Backpropagation through time)
• Part-of-speech tagging
• Conditional language modeling
• Neural probabilistic language models

Machine Translation and Language Generation

• Translation task
• Seq-to-Seq models
• Attention mechanism, Self-Attention
• Transformers
• Decoding, Graph search, common strategies

You'll learn about PyTorch, a machine learning framework used for applications such as computer vision and natural language processing. You'll also learn about HuggingFace library, and 
use these tools to develop code for your assignment. 

Learning outcomes

By the end of this course you'll have:

• learned about the core foundations of deep learning applied to NLP.
• have obtained a solid understanding and ability to build systems (in PyTorch) for some of the major tasks in NLP. 

Assessment and credit

The assessment will be based on coursework programming assignments covering both theoretical and applied aspects of deep learning and its intersection with Natural Language Processing.

The assessment will: 

• include a programming assignment covering a fundamental task in natural language processing (language modeling).
• involve submitting a coding and a report.

The coding will give you hands-on experience, while the report writing aims to assess your ability to describe your work in the scientific format.

This should include a theoretical explanation of the models used, laying out the experimental setups, and evaluating the models’ performance. 

If successful in the assessment, the course offers 15 credits which can be accumulated towards a Master's degree. 

Who this course is for

People who are familiar with basic deep learning concepts.

Required reading

The recommended textbooks for this course are Natural Language Processing by Jacob Eisenstein and Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville.

Course team

Dr Ilija Bogunovic

Ilija is a lecturer in Machine Learning Systems Engineering in UCL's Department of Electronic and Electrical Engineering.

Course information last modified: 22 Sep 2023, 16:55