What on Earth did we get Ourselves into?

Metaballs, chaos equations, predators and their prey. Contemporary dance, neurological time travel, memories of the future. Potatoes. What does all of this mean? To explain it all to you, we’ll need to travel a few steps back in time, into May 2019…

What’s this Cool-Looking Competition Thing?

In April 2019, a buzz from the tech field reached our team member Saara. She had heard through the grapevine that the Finnish National Opera and Ballet was breaking new ground, and planning to do something unprecedented: a new, immersive opera experience. And they were looking for people to do it with.

Our team assembled organically: a mix of designers, developers, and artists. Between us, we have experience in machine learning, computer imaging, service and graphic design, performing and visual arts. We shared a vision of a digital experience that would incorporate our specialties, creating a harmonious world that would awaken the emotions of its visitors.

The First Session competition included two prompts: a story entitled “Memories of the Future” from artist and dramaturge Paula Vesala, and musical work by the composer Esa-Pekka Salonen. Both prompts underlined the concept of time as non-linear: how we can neurologically time travel within our minds. We were to create a concept that reflected the themes of these prompts.

After several group meetings and a few long nights, our pitch was as follows:

The visitor enters an immersive digital world of constant change,

and shapes its future through encounters with others and their memories.

3 Days, 2 Mentors, and 7 Exhausted Team Members

To our pleasant surprise, our team was selected to take part in the First Session – the competition where the winning concept would be chosen. We were in very good company: the competition had received 195 applications from 32 different countries! Over the next two and a half days, we put our heads together and hustled to develop our concept further.

Opera Beyond itself was a doozy. We were excited to meet so many talented colleagues and judges, and sad to realize we did not have time to mingle with them. We spent all our time refining our concept: hammering out our storyline, and what we wanted to say with the piece. For most of us, the worst scenario would be a lame concept. Something that did not leave the visitor feeling anything. 

During our first feedback session with Esa-Pekka Salonen and Paula Vesala, we received some critical feedback. The musical and acoustic dimensions of our concept were weak. It made sense: none of us were professional musicians or sound technicians. We went into crunch time: time to make it work!

We used the phone-a-friend option a few times. We tried interpretive dance to get our brain juice flowing. By the end, we were all tired, having spent the early morning hours furiously typing and editing our pitch video. We were giddy in the morning, eager to present our idea to everyone else at the First Session.

It was time to pitch.

An Immersive World where Organic and Artificial Life Meet

Our renewed concept was built on the idea that users would be able to enter a world where the artificial was made tangible. Artificial intelligence would be personified as agents, which behave and interact with the user in distinct ways.

Initially, we were inspired by predator-prey models. In these models, agents have drives which facilitate their behaviour, instead of being hard-coded. A predator is inclined to chase its prey, while the prey attempts to avoid the predators and take refuge in groups of its own kind. Expanding on this, we could build models for all kinds of distinct AI agents: we could build a whole ecosystem.

The potential visual and movement patterns of these agents was inspiring for our team – their floating through space felt like a rhythmic dance. Our professional dance artist Iina was interested in using motion capture – in order to lend organic movement to the artificial agents. The agents could exist in the form of metaballs: non-euclidean shapes that are defined according to mathematical equations, but with organic movement. We wanted to envelop the user in a world of digital wonder. 

We pitched a spherical space, where the movement of the agents could be examined by the user from multiple angles. If the users felt like they were part of the agents’ world, they would be encouraged to interact with them – and realize that they could affect the agents’ behaviour with their own. Our team’s technical specialists (Joonas and Heikki) suggested projection mapping as the best technology to accomplish this effect. It was also important to consider how the interaction would work. Our team agreed that the interaction should feel organic, and like the user was actually at the world. We agreed on a goal to avoid all extra equipment such as headphones, if at all possible. 

Narrative wise, we were inspired by the concept of non-linearity of time, and especially the chaos equation. The equation posits that microscopic changes, such as the flap or a butterfly’s wings, can create a rainstorm on the other side of the world. This succinctly describes the nature of the world (both organic and digital): there is beauty in chaos.

Out of these thoughts, we came up with our concept “Echo”. Echo would reflect the opera of the future: it is participatory. It brings the Opera and Ballet among people, and enables connections with human and artificial minds.

Up Next: Potatoes and Church Halls

Our concept was selected to be developed further. All the pterodactyl screaming ringing in my ears seemed worth it now.

We formed a proper collective, taking the name of the concept: Ekho. The name refers to the Greek goddess Ekho, who only spoke when others spoke, and didn’t when others didn’t. We felt this described our team. 

So, what are we doing next?

The Opera’s staff are helping us realize our shared vision, building a cathedral shaped hall for us to physically test our concept in. Our original concept of a sphere-shaped space was modified to a church-type hall for the sake of testing, in order to accommodate for more users using the space simultaneously. We want to encourage the users to interact with not only the artificial intelligence agents, but each other.

To test out user interaction with the concept, our contemporary dance expert Iina had prepared a workshop for us. The workshop involved pushing potatoes around on the floor with whatever body part felt natural, and seeing how you and your potato interact with other potatoes. While it sounds silly, this very effectively demoed the relationship one forms with their own potato (in our concept, their digital reflection), and other people’s potatoes. A lo-fi prototype is always effective before testing out something fancy, technical, and time consuming. 

We will continue in the same vein: building our storyline with Paula Vesala, and the musical score with Esa-Pekka Salonen. I’ll continue writing about this ongoing madness and beauty, so stay tuned.

Feel free to reach out to us on Twitter: https://twitter.com/EkhoCollective
Follow us on Instagram: https://www.instagram.com/ekhocollective/
You can also reach out to the author of this blog post: https://twitter.com/minjaaxelsson