1
Experimental museology: immersive visualisation and cultural (big) data
Sarah Kenderdine
In 1889, Smithsonian Institute curator George B. Goode (1891, p. 427) delivered an anticipatory lecture entitled âThe future of the museumâ in which he forecast that the museum would one day âstand side by side with the library and the laboratory.â As public cultural institutions, the primary mission of galleries, libraries, archives and museums is to provide citizens with knowledge not only about but through their collections and cultural heritage materials. With the advent of the Internet, from the mid-1990s the opportunities emerged for websites of public collections to become a virtual counterpart to the physical museum (Cameron & Kenderdine, 2007; Kreiseler et al., 2017). Much has been made of the democratising potential of the digital transformation of museums (Museums and machines, 2016; Taylor & Gibson, 2017). Paradoxically, the mass digitisation of public collections and their vast unseen annals, along with the concomitant metadata, has brought about an information overload that not only defies curation but also arguably further submerges the meaning of the archive in its own data (Vesna, 2007).
In recent decades, museum commentators have hinted that visualisation is a crucial intermediary between the digital archive and its big data, functioning both within galleries and beyond their physical location as networked access. A brief review of online cultural heritage collections reveals a visualisation revolution that requires rethinking the operational framework and the role of the museum in society (see, Windhager et al., 2019). At a deeper level, as Cui (2019) points out, information visualisation has itself altered how we view databases. Nonetheless, a large gap exists between what a human can do with data and what a machine might do. While this problem is often described in terms of a scalability challenge for visual analytics, in reality both human and machine limitations are at the root of this fundamental issue. Creating greater public engagement with collections through visualisation is not the magical solution for the problems facing museums â it is one step in a revolution of the way in which stories are told, and narrative unfolds.
While, on one level, visualisation is regarded as a simple means of communication, of a one-way information transfer, the critical frontier of advanced analytic tools, visualisations and situated interfaces are those that can bring audiences into meaningful communication with and creative co-production of cultural heritage. The museological turn toward a humanistic ethos has hardly been rapid. Interrogations began as far back as the 1980s, with the application of post-colonial critique to museums (see, Bennett, 1995, 2004), which occurred in parallel with the proposition that it might reinvent itself as the ânewâ museum (Vergo, 1989). Since this time, museological and curatorial domains have been serially re-born as participatory, responsive, reflexive, inclusive, interrogative, relational and activist, with varying degrees of real structural change (see, Abungu, 2004; Butts, 2002; Chipangura & Chipangura, 2020; Coleman, 2018; Mithlo, 2004; Vawda, 2019).
Experimental museology not only embraces this constantly changing landscape, it also challenges the mentality that feigns to âopen upâ the museum through digitalisation while leaving intact its outdated, linear and canonical ethos as the chief custodian of heritage and authority on history. My own work in the field of experimental visualisation has made a departure from these institutional orthodoxies, as it has sought to transform public engagement with heritage through the application of aesthetic practice to cultural (big) data and the design of novel interactive frameworks. One of the earliest systems I created, The Virtual Room, was realised for Museum Victoria, Australia. Designed as a permanent gallery for situated experience in 2003, this stereographic interactive and immersive environment was one of the worldâs first large-scale visualisation systems for the mass public (see, Kenderdine & Hart, 2003). I then went on to collaborate extensively with the iCinema Centre for Interactive Cinema Research at UNSW Australia and then to lead two research laboratories in the domain: the Applied Laboratory for Interactive Visualisation and Embodiment (ALiVE), Hong Kong, and UNSW Sydneyâs Expanded Perception and Interaction Centre (EPICentre).
In 2017, I established the Laboratory for Experimental Museology (eM+) at Ăcole polytechnique fĂ©dĂ©rale de Lausanne (EPFL) in Switzerland (Laboratory for Experimental Museology, n.d.). eM+ combines research from scientific, artistic and humanistic perspectives and promotes post-cinematic, multisensory experiences using experimental platforms. Its location in a 1,500-square-metre warehouse is home to nine large-scale visualisation systems, enabling transdisciplinary research at the intersection of aesthetics, immersive visualisation, interactive narrative and cultural data.
Despite such ground-breaking work, the expansion of the museological realm into the rich sensory, perceptual and social potential of experimental visualisation remains unchartered territory for many museums. This impasse was nowhere more evident after collecting organisations in 2020 around the world were closed to publics during the COVID-19 crisis, museum and gallery curators, directors and collection managers have been prompted to fling open the portals of their archives online. That a plethora of âvirtual visitsâ have plunged us into emptied gallery spaces illustrates just how many institutions have failed to fully understand the needs and desires of their audiences. In an era of networked digital culture, as set out by Hull and Scott (2013), many members of the public are able and ready to exploit the creative and participatory opportunities via the combined affordances of digital archives and social media.
Mapping out a possible path for such a future museum, this chapter elucidates some of the innovations in visualisation that I have developed in the domain of experimental museology, categorised as three approaches: collections visualisation, embodied visualisation and spatial and temporal visualisation. Before doing so, I provide here a brief overview of the state of the art of visualisation in the cultural heritage sector, the context out of which my own expertise continues to evolve.
Visualisation and experimental museology
Mapping data to visual representations has been used for centuries to reveal patterns, to communicate complex ideas and to tell stories. For Leonardo da Vinci, visuality in painting was the paragon of apprehension, surpassing both poetry and music. Daniel Albright (2014), in his theory of âpanaesthetics,â examines the way in which one art form can be translated to another (e.g. a painting is transformed into a musical composition).
Image-making for Harald Klinke (2014, p. 5) is not a âsimple process of externalisation of internal pictures â the process of drawing and painting [is] central to the process of thinking. It is not perception alone, but the complex process of picture-making that grasps reality and gives ideas about the world some sort of order.â At various junctures in the discourse of the humanities, scholars have pronounced new notions about how humans constitute reality. Both W.J.T. Mitchellâs âpictorial turnâ (1994) and Erwin Panofskyâs âiconologyâ (1939) are theories that focussed on images rather than language. Ernst Cassirer, on the other hand, characterised images as âgiving sense to the world by symbolising ⊠experience in a process of perception and representationâ (Cassirer quoted in Klinke, 2014, p. 6). As such, and as Klinke contends (2014, p. 6), âthe question of images and their epistemic content ultimately points back to the human, who perceives, imagines and creates pictures. ⊠The power of images stems not from the images themselves, but from humans, who give them meaning.â
âVisualisationâ encompasses these theories of the image and the function of their creation as a cognitive, transformative act. Visualisation, in the words of Scagnetti (2011), can be described as âa medium for communication (or persuasion, or engagement)â; a tool for understanding (or problem solving, planning, orienting)â; a âvisual rhetoricâ made of objects, including relations among those objects and tools for managing the relation between objects and environment; and as a âvisual epistemologyâ describing how we interpret the world. âInformation visualisationâ is a graphical representation of (digital) data specifically designed to harness and augment basic powers of human perception for the task of comprehending large-scale information, and interactive visual representations of data are proven to further âamplify cognitionâ (Card et al., 1999). Suffice to say, in response to digitisation, databases and networks, visualisation is becoming a dominant force through all disciplines. The application of digital visualisation techniques to cultural heritage data sets is today celebrated as a new and innovative research methodology (Bailey & Pregill, 2014).
In defining visualisation as a representation, interpretive and revelatory, visualisation is both science and language. Like science, it represents data accurately and methodically, allowing us to detect underlying patterns, trends and relationships and, like language, it is used to convey meaning. Through visualisation, data is encoded into symbols and thus forms a system of semiotics. And yet visualisation poses specific problems for knowledge production.
As design and humanities scholar Johanna Drucker points out in the preface to her book Graphesis (2014), the reader of visualisations must learn the conventions of the diagrammatic knowledge form as this syntax is not inherent. These forms of visualisation may be infinitely varied and/or highly specific. In other words, graphic inscription itself is defined by characteristics that makes it hard to analyse. Unlike language, it is not a system that has a stable code, this makes visual analysis and visuality different from linguistics and language-based notational systems. Furthermore, images may conceal the decisions and processes on which they are based and appear to simply represent âknowledge.â Visual representation reveals what is at stake in the distinction between information and interpretation within humanities practices. Drucker (2014, preface) argues that âgenerative,â âdynamicâ and âdiagrammaticâ images produce knowledge and that visualisations constitute information that possess the same legitimacy as any other human expression, such as written text. For Drucker, visualisations are âgraphical forms expressing interpretationâ (2014, p. 54), and that because of the âfundamentally interpreted condition on which data is constructedâ (2014, p. 129) visualisations are a feature of both âknowledge production and [its] presentationâ (2014, p. 69).
The visualisation of cultural heritage collections began in the mid-1990s, as humanities research sought to expand the possibilities of descriptive and analytic data in object notation, metadata and the standard âsimple searchâ interface or inventory. On one hand, the constraints of relying on collection metadata as a search tool were immediately evident, it being uneven, unfinished and sometimes subjective. On the other hand, information retrieval itself, as underscored by Rogers et al. (2014), has presented serious limitations as a model for meaningful public engagement with cultural heritage collections.
Visualisation as an experimental approach is nonetheless unique in its potential to support all kinds of informal learning spaces. Cultural heritage collections comprise a potentially vast array of encounters within varied institutional settings, and experimental applications of visualisation have the potential to open up the museological realm to exploration both within and beyond the institutional walls. As outlined by Falk and Dierking (2019), the first methods to enhance the understanding of data through new visualisation literacies arose from research partnerships between science museums and educators, often with a specific focus on visual literacy or analytics as an enabler for scientific learning, study, or knowledge transfer (i.e., Kenderdine et al., 2016; Lock et al., 2018; Moss, 2019). Along these lines, visualisation has been developed as an effective tool for education, journalism and research knowledge transfer. Yet, when a visualisation is presented within the confines of the museum itself it is often curtailed in its capacity to respond to or collaborate with audiences, due to the fact that it is often employed to facilitate one-way communication, civic education or audience survey.
In contrast, creative approaches to interactive aesthetics and design have taken more inventive turns in the hands of artists (Jacobs et al., 2016). Countless examples of this rich field are documented in the proceedings of the Electronic Visualisation and Arts conferences (n.d.), which emerged from the Computer Arts Society, established in 1969, demonstrating the extent to which artists have been innovating throughout the entire modern history of computational science. Arguably evolving from media artistsâ initiatives, unique research partnerships have flourished in recent years, uniting creatives with educators, technologists, entertainment and gaming industries, heritage professionals and cultural institutions. These interdisciplinary alliances have fostered a burgeoning of innovation in sensory experiences via the creative exploration of multimodal technologies and dimensional realities, interface design, interactivity and data visualisation (Cantoni et al., 2019).
The domain of digital humanities em...