Close
Touchless Computing: UCL MotionInput 3
What is UCL MotionInput?
UCL MotionInput, now in its 3rd generation, is a major software package launched globally for touchless computer interactions of nearly all ranges of human movement using just a webcam and PC.
The platform allows users to control and interact with a computer using physical gestures captured by their web camera. It replaces input functionality normally provided with a keyboard, mouse or joypad, and instead uses movements of their hands, head, eyes, mouth, or even their full body joints.
How UCL MotionInput started
UCL MotionInput was first created in response to the global COVID-19 pandemic.
As lockdown forced staff and students away from campus, UCL launched a range of initiatives to make teaching remotely work for all. In particular, UCL began conversations with the UK’s NHS, Intel, Microsoft and IBM about possible additions for the NEWS2 protocol, a system used by the NHS to triage patients quickly, but required physical touch and examination.
At the same time, Machine Learning technologies such as Tensorflow, and Computer Vision techniques such as Convolutional Neural Network (CNN) Training models were accelerating in their development.
This led to UCL MotionInput being proposed as a final year project, through UCL Computer Science’s Industry Exchange Network (IXN), in September 2020.
Version 3
The 3rd generation of MotionInput now enables a wide range of combinations of human movements, also with federated and offline speech processing for live captioning and speech commands, for existing applications. It has several orders of magnitude improvements in efficiency and has an architecture for the supporting growth of touchless computing apps as an ecosystem. It is also starting the journey towards building for multiple platforms.
What are the benefits?
Any screen is now a multi-touch touchscreen
Multi-touch touchscreens can be used anywhere, in schools and hospitals as well as at home.
A lecturer can control PowerPoint without reaching for the laptop. A chef can navigate pages without touching the screen. A surgeon can review and annotate CT scan images, without needing assistance to navigate the computer.
Facial navigation
Facial navigation enables a greater degree of accessibility by letting a user navigate either with their nose, if they have spinal conditions, or with their eyes for more serious conditions.
The user can mix this with various keywords and customise their own applications with shortcut phrases.
For example, saying “click”, “right click” and “double click” does those actions. The same can be triggered by using facial features such as smiling, raising an eyebrow or opening the mouth.
VR without the VR
It enables you to control your existing Windows games by movements and speech, either by mimicking your game control pad with your two hands, or by setting hit targets to hit in the space around you.
It puts you into your games, where you can walk on the spot as your game character to move and respond with actions within your games.
In-air keyboard and drawing
For industrial settings, it will let you type in a virtual keyboard, or draw and annotate with digital inking and depth, in the air.
Live captioning, dictation and phrase shortcuts
It can let users make use of live captioning and call out phrases for keyboard shortcuts, in any existing Windows applications and games. The phrases can be defined by the user.
Who is involved in the project?
Academic
UCL MotionInput 3 has the following project academics:
- Prof Dean Mohamedally (Project lead and Director)
- Prof Graham Roberts
- Mrs Sheena Visram
- Dr Atia Rafiq (honorary)
- Prof Joseph Connor (honorary) at UCL
Industry
- Prof Lee Stott (Microsoft)
- Prof John McNamara (IBM)
- Prof Neil Sebire (NHS/GOSH DRIVE)
- Prof Costas Stylianou (Intel)
- Prof Phillippa Chick (Intel)
- Cathy Cummings (Intl. Alliance of ALS/MND Associations)
Senior directors, engineers, designers, and testers from Intel, IBM, Microsoft, Google and the NHS (including GOSH DRIVE) have participated in several years of development at various stages with the students, engaging them, enabling them to present their findings and giving them feedback to refine the solutions in the project.
UCL Students
University College London students, as part of the UCL Industry Exchange Network programme:
Version 1 (Summer 2020)
- Lu Han
- Emil Almazov
Version 2 (Summer 2021)
- Ashild Kummen
- Ali Hassan
- Chenuka Ratwatte
- Guanlin Li
- Quianying Lu
- Robert Shaw
- Teodora Ganeva
- Yang Zou
Version 3 (Oct 2021-June 2022)
Version 3.0-3.02
- Sinead Tattan – Lead Student Project Architect (Final Year CS)
- Carmen Meinson – Lead Software Development Architect (Second Year CS)
- Aaisha Niraula
- Abinav Baskar
- Adi Bozzhanov
- Alexandros Theofanous
- Ali Amiri Souri
- Andrzej Szablewski
- Aryan Jani
- Aryan Nevgi
- Ben Threader
- Chris Zhang
- Clarissa Sandejas
- Daniel Rempel
- Eesha Irfan
- Eva Miah
- Fawziyah Hussain
- Felipe Jin Li
- James Zhong
- Jason Ho
- Jianxuan Cao
- Jiaying Huang
- Kaiwen Xue
- Karunya Selvaratnam
- Keyur Narotomo
- Kyujin Sim (Chris)
- Lama Alluwaymi
- Mari Takeuchi
- Michelle Chan
- Oluwaponmile Femi-Sunmaila
- Phoenix Sun
- Pun Kamthornthip
- Radu-Bogdan Priboi
- Rakshita Kumar
- Raquel Sofia Fernandes Silva
- Samuel Emilolorun
- Elyn See Kailin
- Siam Islam
- Sibghah Khan
- Sricharan Sanakkayala
- Thomas Langford
- Tianhao Chen
- Yadong(Adam) Liu
- Yan Tung Cheryl Lai
- Zemiao Huang
Versions 3.03-3.1 (June 2022-September 2022)
- Anelia Gardarzhieva (Lead Architect 3.1)
- Gincarlo Grasso
- Thomas Langford
- Jiahui Shi
- Vivek Vijay
- Elynor Kamil
Version 3.11 Windows Store Build (December 2022)
- Alex Clarke (Lead Architect 3.11 In-Air Multitouch and 3.2 Facial Navigation)
- Joseph Marcillo-Coronado (Compiler and Windows Store Team Lead)
- Nerea Sainz De La Maza Melon (Compiler and Windows Store Team Lead 3.2)
- Anelia Gardarzhieva (Lead Teaching Assistant and MFC Base Code)
- Abriele Qudsi
- Chaitu Nookala
Version 3.2 Full Team (Sept 2022-April 2023)
- Anelia Gardarzhieva and Mohseen Hussain (Joint Lead Architects v3.2)
- Aaryaman Sharma
- Abdullah Ahmed
- Abdurrahmaan Ali
- Abid Ali
- Abriele Qudsi
- Adil Omar-Mufti
- Aishwarya Bandaru
- Ajay Mahendrakumaran
- Akram Ziane
- Alex Clarke
- Alexandra Irimia
- Amir Solanki
- Arvind Sethu
- Aryan Agarwal
- Baixu Chen
- Calin Hadarean
- Can Ertugrul
- Chaitu Nookala
- Chan Lee
- Chidinma Ezeji
- Chris Zhang
- Damian Ziaber
- David Nentwig
- Donghyun Lee
- Dongyeon Park
- Eloise Vriesman
- Fabian Bindley
- Filip Trhlik
- Filipp Kondrashov
- Gauri Desai
- Ghalia Alsayed
- Hadi Khan
- Harryn Oh
- Imaad Zaffar
- Jason Kim
- Jianheng Huo
- Jie Li
- Joseph Marcillo-Coronado
- Julia Xu
- Kaartik Nagarajan
- Liv Urwin
- Luis Rodrigues Vieira
- Luke Jackson
- Maria Toma
- Mateusz Krawczynski
- Minghui Cai
- Molly Zhu
- Nandini Chavda
- Nerea Sainz De La Maza Melon
- Peter Xu
- Pooja Chhaya
- Rebecca Conforti
- Robin Stewart
- Setareh Eskandari
- Taha Chowdhury
- Takuya Boehringer
- Tina Hou
- Vincent King
- Weiyi Zhang
- Yidan Zhu
- Youngwoo Jung
- Zhaoyan Dong
- Zineb Flahy