ETC Student Nina Wang uses State-of-the-Art Facilities To Create 'Flow'

In spring 2024, second-year Entertainment Technology Center (ETC) student Nina Wang(opens in new window) led a team of ETC students on "Flow(opens in new window)" — a one-of-a-kind project meant to utilize the state-of-the-art facilities on offer at the ETC. In fact, the project was specifically conceived with one of those facilities in mind: the CAVERN.

The CAVERN is a room in the ETC featuring multiple projectors synced together to project stereoscopic 3D onto a 20-foot diameter, 270-degree curved screen. Led by Director of Information Technology Steve Audia(opens in new window), the ETC Technology Team designed and built the CAVERN system in 2022 with the goal of creating a space for students to develop multiplayer, in-person interactive experiences.

“Sometimes I introduce visitors to the CAVERN by telling them it’s ‘VR without a helmet,’” Audia said. “To quote one (Disney) Imagineer, the ‘walls go away.’” For Wang, this presented itself as a clear opportunity.

“I always advise my students to pitch their own projects if they want to ensure it’s something they’re passionate about,” said ETC Associate Teaching Professor and “Flow” faculty adviser Ralph Vituccio(opens in new window). “Nina Wang is an exceptionally talented, one-of-a-kind student, and she took this advice to heart. … Nina and her team were able to pitch the ETC faculty a clear, beautiful artistic experience, reimagining a traditional Chinese story into a new interactive media space. It was the perfect marriage of traditional storytelling with emerging mediated narrative technology.”

Wang has danced for over 21 years, something evident in the emphasis on movement that defines “Flow.”

“I thought of creating an art piece involving dance from a long time ago,” she said. “Then when I came to the ETC, the idea of creating a project combining light, shadow, color, dance, music, movements and interaction emerged.”

The experience begins with a meeting between a deer and a dancer — following them as they grow up, befriend each other, and encounter challenges that can only be overcome through the audience interacting with the screen through motion. The dancer’s movements were choreographed and performed by Wang herself in CMU’s Motion Capture Lab(opens in new window), providing a foundation not just for its character animation but also for the audience movements tracked by motion sensors throughout the room. Guests have to gesture with their arms similar to the on-screen choreography for the story to progress — an intuitive way of introducing guests to the distinctive, graceful style of traditional Chinese dance.

“Flow” was the first CAVERN project to experiment with using sensors for body-based control, which means the development process included plenty of trial and error. Wang and her team — fellow ETC students Ashling Tu, Christine Jung, Laura Yang, Shih-Hung Liu and Jiajian Zhang  — initially installed only one body tracking sensor and provided guests with more indirect control and little instruction. Over the course of the semester Wang’s team invited guests to more than 70 demo, playtest and showcase sessions in the CAVERN, improving it each time based on their feedback.

The team installed two more sensors to help track the movements of guests and integrated instruction into the game in an intuitive way — adding a live host and using visual effects and sound to help guide their attention.

“They were very dedicated to the principles of user testing, in that they would stand in the corner and not say a word when the players got to the part of the experience where they had to take initiative,” Audia said. “It’s very, very easy to just whisper ‘hey, just move your hands now’ from the back but they held steady, and I think this discipline is a huge part of their success.”

In fact, “Flow” has been such a success it’s continued past the end of the spring 2024 semester. “Bringing ‘Flow’ into the world and making it flow beyond spring 2024 was the primary goal from the very beginning,” Wang said.

The project is now used to demonstrate the CAVERN to people visiting the ETC. When Sony Pictures Entertainment’s Senior Vice President of Advanced Technology Yoshikazu Takashima came to the ETC, he got the chance to try “Flow.”  “Even though I think he was somewhat impressed with the CAVERN itself, I got the sense he was most impressed with the design of the game experience and how well it directed the players without outside instruction,” said Audia, who was there for Takashima’s visit, said.

The project continues to have a life outside of the ETC too. Wang had the opportunity to pitch “Flow” to Zunzheng Digital Video, the Chinese manufacturing partner of Flanders Scientific — a Georgia-based company known for producing high-quality monitors for entertainment professionals like cinematographers and colorists. “After presenting “Flow” to them as well as our plans for the future, they decided to investigate,” Wang said.

In just one week, the team at Zunzheng customized and installed a curved LED wall with the same dimensions and stereophonic features as the CAVERN on-site in their office. Because the screen is built-in instead of projected like in the CAVERN, its color and overall brightness will be greatly enhanced in its new presentation. “Flow” will be on display in Zunzheng’s offices for the foreseeable future, acting as a demo for their cutting-edge screen technology. 

And that’s not all: plans are in the works for “Flow” to be exhibited at art centers and museums such as Sea World Culture and Arts Center(opens in new window) at the end of this year. 

“As the director of this project, I can’t express enough how much ‘Flow’ has given me. It led me to so many possibilities and taught me more things than I could ever expect; it connected me with industry experts who were invited to see our work, and brought together a team that is now like a family — one that loved and supported each other along the way and will continue to do so,” Wang said. “‘Flow’ will not end after this coming exhibition. It will definitely act as the blueprint for the future. ‘Flow’ keeps flowing.”