Multi-modal Personalisation in Long-Term Human-Robot Interaction

Abstract

While short-term interactions with robots benefit from the novelty effect, long-term interactions can suffer from a decrease in user interest and engagement. Based on the previous research within Human-Robot Interaction (HRI), the thesis presented here is that user experience in long-term human-robot interactions can be improved by personalising the interaction through recognising users and recalling previously learned information.

User recognition is the first step towards personalising the interaction, however, there does not exist a reliable user recognition method for fully autonomous user recognition in long- term HRI for real-world applications. Correspondingly, this thesis proposes a Multi-modal Incremental Bayesian Network (MMIBN) model, which combines face recognition with soft biometrics and allows continuous, incremental and online learning of users, without the need for any preliminary training. We validated the robustness and reliability of this approach with a long-term (4-weeks) real-world study with 14 users and an artificially generated multi-modal long-term user recognition dataset with 200 users.

Following on from this work, we explored personalisation of the interaction in service robotics and socially assistive robotics, based on earlier evidence for the impact of personalisation on long-term interactions. We created the text-based Barista Datasets that contain simulated generic and personalised dialogues for interactions with a barista that recall and suggest user preferences in subsequent interactions in a coffee shop. Based on these datasets, we designed fully autonomous barista robots with MMIBN, automatic speech recognition (ASR) and a rule-based dialogue manager, and evaluated these robots with a real-world long-term (5-day) study with 18 non-native English speakers. The study demonstrated that personalisation mitigates negative user experiences that arise from unreliable speech recognition and the inflexible structure of the rule-based dialogue manager. Consequently, we explored the potential of the state-of-the-art data-driven dialogue models based on the Barista Datasets. The results showed that while data-driven models perform remarkably well in generic task-oriented dialogue, no model could perform sufficiently well for personalisation in long-term interactions.

Lastly, to demonstrate the real-world bene ts of long-term HRI, we design a personalised robot to improve user motivation and adherence to the cardiac rehabilitation programme, and evaluate with a study that ran for 2.5 years at a hospital in Colombia. The robot individually tracked the patients’ health progress and attendance throughout the programme, and provided personalised and immediate feedback based on continuous monitoring for 18 weeks. While the study could not be completed due to the outbreak of COVID-19, our initial findings (with 6 patients) showed that user engagement and motivation for the therapy and adherence were improved and maintained in the long-term interactions.

Overall, the work undertaken provides supporting evidence for our thesis and contributes fundamental stepping stones for future research in personalised long-term HRI to develop robots that can meet and maintain user expectations.