Laila: Ekho Collective’s thoughts

How we created our audiovisual immersive experience, where visitors interact with AI

In my previous blog post talking about our project, I examined how our Ekho Collective’s process of building our immersive installation Laila has changed during COVID-19. Now our project is almost finished, and the tickets are on sale for showings in August 2020 in Helsinki. Here, I’ve collected thoughts from members of our collective on the process of building Laila.

Joonas Nissinen

Joonas Nissinen is our creative technologist with a background in computer graphics and artificial intelligence. He sees Laila as a digital world that balances a human-comprehensible dramatic arc and procedural generation technology:

When creating an operatic experience, we need to have a traditional plot and narrative. By using artificial intelligence in Laila, we’re giving technology its own agency. Introducing agency means that a straightforward narrative is not guaranteed. Currently, we’re working on balancing a chaotic system that has its own will to behave in a certain way, with choreographing parts of that system to fit our narrative. This is difficult.

We’re using AI in Laila in the form of boids, which are computational agents that make decisions based on a simple set of rules. Boids (a.k.a. bird-oids) are biomimetic. They exhibit flocking behaviour, which is present in nature in the form of fish schooling, ant colonies, and of course bird flocking. In Laila, the boids effectively constitute a superorganism: they are a swarm intelligence that exhibits emergent properties. The synergy of agents interacting locally according to simple rules leads to a higher level whole with much more intricate behaviour than that of each individual agent. Through this, Laila becomes a complex system: no one is able to predict everything that will happen.

To solve the problem of choreographing boids, I’ve created a system which allows us to toggle the boids motivations on a scale: how sensitive they are to separation, how easily they align, and how cohesively they move. Additionally, the boids’ environment is made of an ever-changing vector field generated using various pseudo-random noise functions. The field functions kind of like currents in a river — it’s easier to go in certain directions, and harder to go in others. These vector fields can also be used to enhance dramaturgy, for example by encouraging the boids to move in a spiral at a certain point in the experience.

So, our boids follow a variety of behavioural models, and react to the several forces of their world (e.g. gravity). Then, we introduce Laila’s visitors into the boids’ world by tracking them with LIDAR. With tracking, the visitors are given their own agency and field of influence. Boids are assigned to react in specific ways to their “own” visitor. Visitors also introduce changes to the world’s vector field, creating changes to the flow of its currents.

As our boids move through the timeline of Laila and react to this aggregation of influences, butterfly effects are created. This means that initially small changes in the system created by the interaction of the boids may accumulate into larger changes later on. That introduces an element of unpredictability and mystery to Laila.

Olli Kilpi

Olli Kilpi is our motion designer and visual artist, pictured here evaluating the crispness of Laila’s projections. He shared his thoughts on what he’s learned during building Laila, and how he feels about the project at the stage that it is now:

At this point in the project, close to its finalisation, we’re wrestling with balancing the experience’s certainty with the degree to which it generates itself. We want to give the visitor freedom and a role in the story. We also want to impart our own story to the visitor. We’re finding a balance between these two things — how the user will be able to interact, and to what extent they can affect what they experience.

I’ve worked with VR and fine art before, but designing a 360-degree dome brings entirely new challenges. In VR, there are more pixels and precision. However, in 360 we have a person’s full field of vision as our stage — rather than the limited tunnel vision offered by most VR goggles. In VR, the perspective of the user moves along with them, since they are in the experience alone. Here, I’ve built visuals that work for all of our visitors simultaneously, since the space, and thus the perspective, is shared.

An interesting thing about this project is that since this is a new field of design, there are no existing rules. I’ve realised that in a 360 setting, the traditional rules of composition that exist on a flat page do not apply. This environment is more natural, like walking in nature — it is not a snapshot of an environment, or an environment viewed through a certain lens, but rather an environment in and of itself.

Saara-Henriikka Mäkinen

Saara-Henriikka Mäkinen, our experience and visual designer, talks about what she’s learned about Laila during user tests:

Since we’ve crafted Laila to react to its visitors, it’s intriguing to see how many levels of reshaping our visitors bring to the work. Each person has their own personality, memories, and preconceptions, which all affect their interpretation of the work. These things also affect how they interact — with both the work and the members of their respective visitor group. For example, one of our test visitors remarked that “I didn’t move, since nobody else moved either”. Another remarked that “I’m that person who wants to break up groups — I want to move independently.” Evidently, some see Laila as a playing field: they start controlling or even dominating the experience. Then again, some visitors don’t want to be noticed, and would rather disappear within the work.

Since they contain different types of people, each group has a unique experience — on both a mental and concrete level. For example, if two friends join different groups and then discuss Laila, they may not recognize everything the other is talking about. Since group dynamics affect which visuals and sounds are produced, the other group’s visitors may not have experienced a specific section.

Through Laila, we want to offer possibilities for people to have insights about the work, together. They learn how Laila interacts by observing each other. In this sense, Laila is both a personal and a shared journey.

Minja Axelsson (the author of this blog)

I’m an experience designer and robotics researcher. Here are some of my thoughts on the meanings in Laila:

Laila is a fantasy of our futures — dystopias, utopias, and realism. Our visitors experience all of these possibilities, and participate in their creation. I hope that through this, visitors can get some ownership of the work. Even if the visitor decides to experience Laila as a passive receiver, their presence still affects how the work unfolds. This carries parallels to the real world: passivity is also a choice that carries consequences. I’m a bit of an activist, so I’m interpreting this work through that lens: the choices we make matter.

In Laila, nobody can choose what will happen by themselves. It’s a complex system, built through the interaction of its AI and visitors. The ideas behind Laila have been inspired by chaos theory and the butterfly effect: how a small change in the initial state of a system can create large effects at a later state (i.e. a tornado’s path being influenced by the flapping of the wings of a distant butterfly weeks earlier), and how a seemingly random systems have underlying patterns and interconnectedness. Laila has a lot of parameters, and in theory there is an unlimited amount of things that can happen, within a certain framework. When visitors enter the work, a specific reality is created, with opportunities opening and closing up. It’s related to the Copenhagen interpretation — a certain reality is created when it is measured, and other options collapse out of potential existence.

While Laila is an experience with a storyline in the traditional sense, I think this philosophy is still present in the work. People contribute a voice recording of themselves before they enter Laila, through which they actively take part in building its world. I hope this gives people the feeling that their voices matter, and are incorporating themself into a complex whole. I’ve enjoyed hearing from visitors that they’ve experienced the work as meditative. I see this as evidence that it gives space for them to stop and think. I’m interested to see what thoughts it sparks about our humanity, technology, their relationship, and what kind of future we will build together.

Essi Huotari

Our AI interaction designer Essi Huotari discusses the role of storytelling in designing AI interactions for Laila:

I view Laila as having three fundamental actants: the story, the AI, and the visitors’ interactions. My biggest design challenge has been balancing our control of Laila’s story with power given to the AI and the visitors to affect this story. Alternating between having and losing control is the driving force of our work.

I think what defines storytelling is the significance of time: a timeline, even an abstract one, structures a story, and gives it intention. Certain things happen after other things. Since Laila has a storyline, we sometimes need to grasp control from the visitors and the AI to move the story forward. This separates Laila from many other interactive installations, which function more like playgrounds with a given set of rules always in effect. Finding the correct places to intervene is the real challenge. I think telling a story and giving a visitor full control are two ends of a spectrum. Laila moves along this spectrum during its storyline.

I think the most intriguing part of the design process has been making space for experimenting, and making use of happy accidents. It is fascinating to design for an experience with AI agents that have agency of their own. Instead of choosing predetermined actions, I have had to search for correct combinations of parametres, goals and laws of physics within Laila that result in the AI behavior that we are looking for. As a designer, I’m used to making decisions about the specific look and feel of finished work. Here I have to relinquish some of that control. In return, we have discovered delightful AI behaviours and interactions that we couldn’t have imagined beforehand.

Iina Taijonlahti

Our choreographer Iina Taijonlahti is fascinated by how visitors move within Laila, and talks about its meaning within the work:

Essentially, a social choreography is created by the visitors within Laila. Laila carries the same conventions as the outside world. Even in a grocery store, people are creating choreographies without realizing it. The way people move depends on a wealth of parameters: whether you know someone, what culture you’re from, and what equipment you move with. Another interesting thing is that depending on the perspective of the visitor, they can act as a member of the audience, or a performer within the work. It’s playful: am I being observed, or am I the observer? In a way, the curtains through which visitors enter Laila act as an entrance to a stage.

I view the space that Laila is built inside of as highly technologically advanced. Laila’s sphere is very unconventional and immersive, as a result of seeing things in 360 degrees all the time. Visitors have commented that they lose their sense of time and place.

An interesting thing about Laila is that we’re giving agency to actants that are not human. This is done through Laila’s AI agents — its swarm intelligence. In this way, Laila is related to the concept of post-humanism, since it is concerned with non-human perspectives. Questions I’ve been pondering are: How can we let go of the ideals of rationalism and humanism, where human intelligence is of central importance? How can we give space to other senses, and new ways of experiencing and distributing power?

Heikki Heiskanen

Our media technology engineer Heikki Heiskanen examines his experience creating Laila:

Working with so many talented people from different fields — such as software development, visual arts, sound design, and opera — has been enthralling. Since Laila is a new type of artistic concept, it has required knowledge from diverse domains. As we’ve integrated this new information and knowledge into Laila, its concept has evolved along the way. This type of working process, where visions from multiple viewpoints come together and are incorporated into one coherent whole, includes its own unique challenges and triumphs.

As we’re preparing to open Laila to visitors, I hope it will be seen as an example of the storytelling capabilities of immersive and interactive technologies. The soundscape is amazing and the visuals are highly imaginative — I’d describe Laila as a magical experience.