Acting On The Edge: Developing Mixed Reality Heritage Performance Through Real-Time 3D And Virtual Production Technologies

Abstract

The past few years have seen a surge of interest for in-situ Mixed Reality (MR) in the field of cultural heritage with engaging and entertaining experiences that enable and enrich learning while augmenting the historical context. This paper contributes to this body of work by introducing a novel application of MR in heritage sites that fuses see-through Augmented Reality (AR) smart glasses with live theatre. We developed two artistic-educational experiences for two heritage sites in the US and the UK exploring the fusion of theatre with historical research as a tool for decolonisation. Our design approach introduces a new interaction design and production model for developing shared location-based MR experiences that is based on dynamic, real-time 3D rendering akin to virtual production processes. Our world-as-a-stage metaphor further draws on the theatricality of the MR medium. The system design consists of a timeline system, an IoT infrastructure and a cross-platform network communication protocol to enable real time interactions between actors, audiences and holographic content. We carried out empirical evaluation studies on both sites that demonstrated the success of the application in engaging participants with the presented histories of the historic sites and its usefulness as a learning tool. Based on the evaluation insights, we discuss interaction design considerations and analyse the proposed model’s added value in establishing a real-time virtual production development and performance environment that can streamline XR prototyping for shared in-situ MR experiences.