Opera is all about singing, so what do you do with a character who is non-verbal?
That's the challenge faced by composer Paola Prestini with her new opera Sensorium Ex, which premiered in late May in Omaha at the Common Senses Festival.
" How do you bring a non-verbal character to life?" Prestini asked. "You know, how do you bring a non-verbal character to life in an opera where everything is about voice?"
Prestini was collaborating with librettist Brenda Shaughnessy, who has a non-verbal son. Together, they developed a dystopian, sci-fi story with one character, Kitsune, who has a disability rendering him unable to speak.
Paola Prestini wanted Kitsune to be a full part of the opera, not just a presence on stage, so she approached R. Luke DuBois, co-director of the NYU Ability Project and a composer. She asked him to develop a device that would give Kitsune a voice.
"We both agreed that it would be open source, so that by the time we finished the opera this would actually really be something that anybody in the non-verbal or minimally speaking community could use," said Prestini.
It was a five-year process. For the first two years, accessibility designer Lauren Race queried the non-verbal community about what they wanted from a new speaking device. She asked, "What do you want with this thing? What's wrong with it? What do you love? What do you hate?"
"There's this mantra that everybody in this community uses," said Prestini, "which is 'nothing about us without us.'"
After the research phase, DuBois and his team set about developing sophisticated technology that uses artificial intelligence and vocal sampling to create an expressive voice.
"This thing is a neural vocoder, which is a posh way of saying that it is a little bit of machine learning that is simulating or recreating the human vocal model," DuBois said. "So, the way it works is you give it some sound and it will infer what your body had to do to make that sound."
Prestini cast two non-verbal actors to share the role of Kitsune on alternate days: Kader Zioueche, 17, who has cerebral palsy, and Jakob Jordan, a 23-year-old man with autism and apraxia, a condition which makes it physically difficult to speak.
Some nonverbal people, like Jordan, use an augmentative and alternative communication device, or AAC device, to talk. They type words into the device and a voice reads them.
But the voice sounds emotionally flat – like a robot. The team fed the AI some of the natural sounds both Jordan and Zioueche made, and it created voices for them.
"Even though it might still sound a little stilted, a little synthetic, it's not perfect, it's them, right? It sounds like them," said DuBois "We had this pretty profound eureka moment. It was very emotional in the room, where everybody was like, 'Oh my God,' because it no longer sounds like their off-the-shelf commercial speech synthesizer. It sounds like they're talking."
But for the opera performances, the team also needed to develop ways for both young men to control the flow of the words that they had pre-recorded, so that they could add emotion and emphasis. So, DuBois turned to electronics and MIT robotics engineer Eric Singer to develop new devices.
DuBois demonstrated how it works by demonstrating with three tiny white boxes hooked up to a laptop and speakers. One gesture slows words down – another speeds them up.
"So, these are the sensors that you have on your automobile that ... know when you're about to get hit by another car. They're collision detection modules," he said. "They have to be fast and they have to be accurate because they have to be able to deploy an airbag. But from a musical context, this means they're accurate. And fast enough to be used expressively in a musical context."
Jordan said, "As someone who was not able to fully communicate for the first 22 years of my life, it is mind-blowing to be in an opera and to be here sharing on NPR."
DuBois hopes the technology can be picked up and developed further. "We wanted to use the opera as a test case for something that we can push out into the world and push out to the world in a way that's best in class, well documented, very open," he said. "I desperately want someone to [be] using our thing and make it cheap, make it free, make it universal." To that end, he and his collaborators have been attending conferences to demonstrate the technology.
As for Jordan, he is dazzled by the new way he can communicate. "When I first heard the sound of my voice come to life, a new realization was born," he said through the new device. "Dream the bigger dreams, you know, the ones you dismiss and hide away because they seem impossible."
Jennifer Vanasco edited the audio and digital versions of this story.
Copyright 2025 NPR
