Beyond the audio and visual: UCL Computer Science is transforming how we interact with technology
Imagine receiving a hug over a video call. What foods encourage risk-taking? Can a scent improve your health? The Multi-Sensory Devices group enhances digital experiences with touch, taste and smell.
Having worked on exhibitions for Tate Britain, reimagined the feeling of flying for British Airways and developed concepts for the metaverse, the team are exploring diverse possibilities.
From commerce to health, they are pioneering surprising technological developments.
Engaging with all our senses
When Marianna started her career as a lecturer in HCI, she focused on the standard audiovisual interfaces.
Nonetheless, she encouraged her students to consider all the senses when designing user interfaces.
In 2011, a Marie Curie Fellowship took her to Newcastle University, where she decided to research tactile, gustatory and olfactory experiences.
As HCI had tended to overlook these senses, there were two major challenges. First, there was a lack of suitable devices to deliver the interactions.
And second, a new vocabulary was needed for users to talk about what they touched, tasted and smelled and for academics to record the results.
Recreating touch
Marianna's studies on touch connected her with Professor Sri Subramanian, then based at Bristol University and now leading UCL's MSD group.
Using an array of ultrasound speakers to generate mid-air haptics, they investigated how mid-air haptics could attempt to mediate emotions.
Marianna also integrated her work on touch with virtual reality and designed material for a glove to enable an astronaut to 'feel' through their spacesuit.
As we increasingly socialise online, we miss the ability to touch one another. This has led researchers to experiment with reproducing tactile sensations digitally.
The EU-funded Touchless project uses neurocognition and artificial intelligence to create novel haptics technologies.
Project partners include the MSD group, the UCL neuroscience community, other universities and industry partners.
Marianna explains more: "Agency, the feeling that you act, bonding, the feeling that you belong, and attachment, the feeling that you care, are all essential for social functions. They can also be expressed through touch.
We're exploring agency, bonding and attachment and their relationship to touch. We want to recreate different tactile sensations for distinct social interaction scenarios, which includes adding virtual touch to video calls."
Another collaborative project is the Textiles Circularity Centre (TCC), funded by UK Research & Innovation. Led by the Royal College of Art, UCL partners include the IOE, UCL's Faculty of Education and Society, UCL Brain Sciences and UCL Computer Science. More partners include another six universities, industry and the public sector.
The TCC aims to support the UK's textile manufacturing and SME clothing companies while reducing reliance on imported and environmentally and ethically impactful materials.
The MSD group's contribution is to harness emerging technologies to relay multisensory experiences. For example, they are exploring how fabrics feel and how that could be translated into mid-air haptics. This means we could 'feel' textiles when online shopping.
The importance of smell
Inventors have endeavoured to use smell to create new experiences for centuries. In the 1860s, theatres and music halls tried to distribute scents to the audience. Cinemas first trialled Smell-O-Vision and AromaRama in the 1950s. All attempts failed dismally.
When Marianna started experimenting with smell, she found only gadgets on the market. There was nothing sophisticated enough to mix fragrances or measure the effect on the individual.
So, using an ERC starting grant, Marianna and her team built their own portable scent delivery device. This led to forming OWidgets, a spinout company from UCL and the University of Sussex.
Through their academic research and OWidgets, the team have investigated wide-ranging applications. They've delivered scent as a component in virtual reality storytelling.
They've tried using smell as a communication channel too, so while the eyes and ears stay focused on a task, scent can convey another message to the brain.
These studies have included using smell in cars to alert drivers if they exceed the speed limit or run low on fuel. They also created olfactory notifications for Slack, the business messaging system.
Marianna has another role as Deputy Director (Digital Health) for the UCL Institute of Healthcare Engineering. Her research into smell is helping in the health and wellbeing sector.
The technology is used to train people who have lost their sense of smell and to test the effect of different scents on body image perception.
A loss of smell can identify the early onset of diseases such as Alzheimer's and Parkinson's. The Rockefeller University in New York is using OWidgets' technology for smell testing for these neurogenetic conditions.
Marianna says: "First of all, people are sceptical about relying on their nose. But it is more important than we think. The contribution that the human sense of smell makes to quality of life and wellbeing is immense.
It's important for food, for hygiene, for security, for social interactions, as well as having a unique link to emotions and memories."
The influence of taste
Taste is difficult to research. It needs a large degree of acceptance from the user as they must open their mouth to participate.
Studies also take place in a laboratory using a gustometer, an intrusive device that inserts tubes into a person's nose or mouth.
But this hasn't stopped Marianna's team from discovering how taste impacts on our emotions and behaviour. In one study, they showed how taste affects our decision-making.
Participants were asked to drink an odourless and colourless solution before performing a task to determine their propensity to take risks.
Each solution carried one of the five basic tastes: sweet, bitter, salty, sour and umami. Studies in the UK and Asia demonstrated that sour tastes consistently promoted riskier behaviour, regardless of existing personality traits.
Marianna says: "Imagine a future where you can taste something in your mouth when you are engaging with an ecommerce advert. We need to be aware that it could influence our decision-making and make us take more risks."
The MSD group has also researched food for space travel. How could it be made more exciting and reminiscent of the food we normally eat?
The advent of 3D food printers offers exciting opportunities to work with taste. The group has also developed a contactless food delivery mechanism.
TastyFloats uses acoustic levitation to suspend morsels of food or drink in the air and bring them to the user's mouth. Ultimately, the group hopes to combine taste and smell to create flavour interfaces.
Rich sensory experiences
Society has long considered smell, taste and touch inferior to sight and sound. But now, there's an emerging desire to examine the complexities of all the senses.
Through social distancing and for medical reasons, the COVID-19 pandemic showed many the harsh reality of a world without touch, smell and taste.
People are also looking to the metaverse: how can an avatar experience the richness of the real world? The MSD group is at the forefront of technological advances.
They are devising a language to describe these new sensory experiences and revealing how digital can bring remarkable interactions.
