Communication skills training intervention based on automated recognition of human emotion and non-verbal behaviour

Abstract

Introduction: Across multiple sectors training programmes aim to help learners improve their communication skills. It is well recognised that non-verbal ‘social signals’ play an important role in effective communication. Previous research in the social signalling domain meticulously observed hours of videos and conducted observational studies to identify these social signals, an approach which is subjective and does not scale with large datasets. Technological developments hold promise to automate observation with possible practical application to training interventions. Objective: Therefore, the aim of the current research is to investigate whether communication skills can be improved using recently developed commercial off-the shelf technology to capture facial expression, voice emotion recognition, hand gestures and honest signals. Methods: Four stages of research were conducted. The first stage was to establish the relevant signals for performance appraisal in media interviews. The second stage was to identify the most appropriate method of providing feedback to trainees that is actionable and understandable. The third stage was to compare whether the designed social signal feedback method was more effective than standard methods of communication skills training in the context of media interviews. Finally, the fourth stage was to assess whether the skills gained in stage three was maintained after 6-months. Performance ratings were collected by an audience who were blind to experimental condition and conversational partners / trainers. Results: Performance ratings collected from the experiment and follow-up stages suggest that the social signal feedback group were more effective communicators compared to the traditional feedback group. The social signal feedback group displayed a significant reduction in frowning in the experiment stage and more positive emotions in the follow-up stage. However, the traditional feedback group exhibited more positive engagement during interviews. Conclusion: The social signal feedback method presented has some benefit over already existing methods of communication skills in media interviews.