Abstract
This thesis investigates whether embodied human expression—captured through facial and full-body motion—can be translated into musically coherent behaviour using time-series machine learning and real-time generative systems. The project develops and evaluates a multimodal framework that links participant-specific emotion inference to an interpretable, artist-configurable generative music system.Across two sensing modalities, personalised BiLSTM-based models achieved ~99%window-level validation accuracy within participants under a strict leak-free protocol, with no evidence of overfitting. These models were deployed within a low-latency real-time inference system. The research introduces a structured translation layer comprising a consensus mapping matrix and a tiered control architecture, mediating between probabilistic inference and musical behaviour and stabilising classifier output through dwell logic, gain calibration, and entropy-informed scheduling. The resulting system operates as a performable generative instrument in which machine learning influences musical development while remaining transparent and adjustable.Empirical comparison between pooled and participant-specific modelling demonstrates that expressive individuality significantly shapes emotion-to-music translation, supporting personalisation as a principled methodological strategy. Geometric analysis and phenomenological reporting further suggest that internally oriented control states, such as Flow, can be learned and rendered musically actionable.Four completed musical artefacts validate the framework under sustained performance conditions, demonstrating that probabilistic emotion inference can support extended musical continuity without destabilising generative structure. The thesis contributes a coherent methodological and technical framework for cross-domain expressive translation, positioning machine learning as a mediating instrument within human creative practice rather than as an autonomous compositional agent.
Awarding Institution(s)
University of Plymouth
Supervisor
Eduardo Miranda, Nuria Bonet
Document Type
Thesis
Publication Date
2026
Embargo Period
2026-10-19
Deposit Date
October 2026
Additional Links
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
Recommended Citation
Mc Corry, M. (2026) Translating Human Emotion into Music: A Multimodal Machine-Learning Framework Linking Movement, Facial Expression, and Real-Time Generative Composition. Thesis. University of Plymouth. Available at: https://doi.org/10.5281/zenodo.21788117
