Home bam

Digital Touch Emoticons: Development, Effectiveness and Usability

A 2017-18 Social Science Plus Pilot Project

  • Project Team

Social Science Principal InvestigatorProfessor Carey Jewitt Professor Technology and Learning, Culture Communication and Media Department/ UCL Knowledge Lab, Institute of Education, UCL

Non-Social Science Co-Investigator
Dr. Aikaterini Fotopoulou Reader in Affective Neuroscience, Research Department of Clinical, Health and Educational Psychology, Division of Psychology and Language Sciences, Brain Sciences, SLMS, UCL

Additional Co-Investigator
Professor Nadia Berthouze Professor in Affective Computing and Interaction, UCL Interaction Centre, connected to both the Division of Psychology and Language Sciences and Computer Science, Brain Sciences, SLMS, UCL

Collaborator Professor Sara Price, Professor of Digital Learning, UCL Knowledge Lab, Department of Culture, Communication and Media, Institute of Education, UCL

  • Research Question

Digital Touch Emoticons will bring together experts on touch from social science,neuroscience and computer science with the aim to understand how affective touch can be digitally-mediated to enhance social communication and positive social feedback. The project will use methods from these three disciplines to generate a prototype design and explore how a device for digital touch communication could be best commercialized in the near future.  More specifically we will ask:

1. What features and specific types of affective touch experience can be translated into socially meaningful digital communicative experiences?

2. What multi-sensorial cues best serve as ‘emoticons’ for such digital affective touch and its associative communications?

3. Which technology best implements digitally-mediated affective touch for socially meaningful communicative experiences?

4. How can qualitative social science methods, quantitative neuroscience methods, design-based computer science and affective computing methods support each other to explore affective touch in the context of digital communication technologies?

  • Focus, rationale and societal relevance

Focus: This project will bring together existing expertise on social, affective touch communication, within the applicants’ fields (sociology, communication, neuroscience, human-computer interaction, affective computing) to explore for the first time, how affective touch can be introduced in mainstream digital communication. It will explore how established digital communicative practices of sending emoticons could be extended to touch (Papacharissi, 2014), to focus on ‘tactile-emoticons’ (1) for social support; (2) admiration (like) and (3) intimacy (love). These capitalize on the recent discovery that touch is the preferred, and the only reliable, non-verbal channel of communication for prosocial emotions (love, admiration), and social intentions such as social support (App et al., 2011; Kirsch et al., 2017; Sauter, 2017).

Rationale: In a contemporary globalized world, social communication is increasingly digitally mediated. The frequent exchange of emotional states and social feedback (e.g. ‘Likes’, ‘Emoticons’) is a significant aspect of communication, often via visual technologies. The absence of touch in digital communication is important as social (affective) touch is a unique aspect of social communication (Hertenstein et al., 2006), the formation of social bonds (Field, 2003), hierarchies (Dunbar, 2010) and of self-other boundaries.

Societal relevance: As optical technologies have expanded and reconfigured the role of the visual in how people communicate, the new wave of digital touch communication devices and environments will exploit and re-design touch practices. The centrality of touch to both human experience and communication underpins the need to better understand the social, psychological and neurological consequences of how touch can be digitally remediated.

  • Research design and methodology

The design has three interconnected phases:

Phase One, will explore touch social practices, norms, and ethics to address Research Questions 1 & 2. We will conduct a literature view, combine our ERC data findings, and facilitate a participatory workshop (20 participants) to identify the features of touch (giving, reception and interpretation) and multisensorial cues that have the potential to communicate distinct emotions in intimate and public digital environments (e.g. social media exchanges).

Phase Two, will inform the design of a wearable device to further explore social digital touch communication (Research Questions 2, 3 & 4). Building on phase one, emerging literature on social digital touch (Huisman, 2017), and design work (Berthouze, 2017), we will explore how to computationally characterize types of social touch to automatically

extract affective cues from touch behaviour and its possible affective meaning (social support, approval, intimacy). We then ask how those meanings could be implemented in the form of tactile expressions (e.g. tactile actuators, sensors, EMG, heat expressions). Methods include ethnographic observation, rapid prototyping workshops, psychophysical assessments of affective touch.

Phase Three, will combine our different disciplinary lenses to explore the social responses and uses of the wearable prototype device (Research Questions 3 & 4). Through observation, user interaction study (50 participants), and psychophysical assessments of affective touch (100 participants) we will examine the features that would enhance the social use of this touch device, with attention to reciprocity, privacy, agency, and controllability. This will enable us to re-calibrate the device towards an exploration of its feasibility and commercial potential.