AI enhanced touchless computing helps users take control of their gaming experience

MotionInput was initially developed by Professor Dean Mohamedally, alongside colleagues and students from UCL Computer Science in 2020 as a means of supporting NHS frontline staff during the COVID-19 pandemic. Since then, it has gone through several generations of improvements, integrating offline and privacy safe Artificial Intelligence for PCs, with the help of industry partners including Intel, IBM and Microsoft. A spinout company from UCL Computer Science, MotionInput Games, was established in 2023.

The AI-enhanced software now includes 24 different features that allow people to interact with a computer in different ways. This includes a user operating the cursor using just their gaze, generating text by writing with a finger in mid-air, leaning in one direction to action a gaming event and hand gestures to perform a range of programmed actions, such as transcribing live text.

The latest functionality has been made possible by artificial intelligence software provided by Intel and the OpenVino toolkit, with thanks for the additional support of devices donated by HP. For example, for users with limb difference or amputees without hands, it can use machine learning models to recognise a user's body and evaluate their motions to correlate to certain actions.

To help test and refine the software, UCL invited BBC technology reporter Paul Carter and pupils from the Richard Cloudesley School in London to give it a go using popular games like Minecraft and Rocket League. It was also tested with Microsoft MakeCode for children learning to code.

This sort of user feedback has been invaluable. It is one of the reasons why the software has gone from strength to strength and is central to the ethos of touchless computing, which aims to be accessible to everyone, driving a new era of equitable computing for all.

Professor Mohamedally from UCL Computer Science, who has led development of MotionInput over the last four years, said: “There are other software and devices that allow a user to control their computer or games console in different ways, but these usually require specialist hardware. MotionInput democratises touchless computing because it’s free to download and you can run it on most laptops and PCs, even older generations of machines.

“We also have explored a wide range of input method combinations now that enable people to play games according to what their bodies can do. We start from the basis of what the user can move with control and work out how easy we can make it to move a mouse or joystick, or to input text. We’re moving beyond touchless computing to equitable computing.”

To date, over 290 UCL students have helped to get MotionInput to where it is today. One of these students was Anelia Gaydardzhieva, who started working on MotionInput as a Master’s student and is now the Chief Technology Officer of MotionInput Games.

Anelia said: “Being a part of MotionInput has been one of the most valuable educational experiences in my life so far. Not only are we working on the latest AI for PC technologies with some of the largest companies in the industry, but we are also contributing to society with our free accessibility software. Seeing the smiles on the faces of children with accessibility needs while playing games with our software has been a truly inspirational experience!”

MotionInput Games has many collaborations planned for later in 2024, including the National Autistic Society and Great Ormond Street Hospital, as well as further work with the International Alliance of ALS-MND Associations.

Links

• UCL MotionInput
• Professor Dean Mohamedally’s academic profile
• UCL Computer Science
• UCL Engineering
• MotionInput Games

Images

• UCL Computer Science students Baicheng Yin, Aleeyah Kadri and Sarah Chaudhary help Richard Cloudesley School student Safiya get set up on MotionInput. Credit: BBC
• Safiya and Paul Carter testing out MotionInput. Credit: BBC

Media contact

Matt Midgley

Tel: +44 (0)20 3108 6995

Email: m.midgley [at] ucl.ac.uk