Good Vibrations: Pitt Undergraduates Create a Device to Help Deaf Kids Experience Music Through Tactile and Visual Feedback

PITTSBURGH (April 4, 2019) … Through the Swanson School of Engineering’s The Art of Making class, an interdisciplinary group of eleven University of Pittsburgh undergraduate students connected with the Western Pennsylvania School for the Deaf (WPSD) and Attack Theatre to create a device that can help hearing-impaired children experience music and express themselves through dance. 

Attack Theatre holds a recurring dance workshop for three-to-six-year-olds at WPSD. The group previously tried using a Bluetooth speaker in a trash can to produce a vibratory effect that the children could touch and interact with, but this design was not kid-friendly and lacked mobility for lessons that necessitate free movement. The Pitt team saw an opportunity to take a fresh look at the problem and design a new system that addresses the needs of both the instructors and the children. However, with no hearing-impaired members, the undergraduates had to find a way to step into the shoes of their end users to better understand their needs.

“This was a profoundly human-centered design problem with multiple stakeholders,” said Dr. Joseph Samosky, assistant professor of bioengineering and director of The Art of Making course. “A new technology, even if it works perfectly, is useless if it isn’t accepted by and accessible to the end user. This team of student innovators really understood and acted on that insight.”

Issam Abushaban, a sophomore bioengineering and computer engineering student, said that the group learned more about their target audience from WPSD teachers. “We discovered that the rhythm of music and the visualization of colors can reflect a certain mood and affect the way that you feel,” he said. “That was something we really wanted to incorporate into our design.”

To better understand the dance element of their task, the group participated in one of Attack Theatre’s workshops catered to deaf and hard-of-hearing children. “A lot of their dance moves were geared toward expressing an emotion, such as stomping to express anger or frustration or skipping to express joy,” said Farah Khan, a senior bioengineering student. “I think this demonstration gave us a different perspective and helped us view music in a new, productive way.”

After completing their background research, the team decided to explore the use of both visual and tactile feedback for their design. They created several early prototypes, including a wrist strap with haptic motors and a disc “floor mat” with multi-hued illumination around the periphery. When the vibrating wrist strap was sampled by the children, the team learned the value of making early prototypes and getting feedback from their users to empirically test design concepts. 

“During our first round of testing, we wanted to pay attention to the reactions that the kids made, rather than focusing on the messages that the interpreter relayed,” said Abushaban. “Some of the kids seemed to be wary or afraid of the wrist strap so the lesson we learned from that meeting was that our product perhaps wasn’t kid-friendly. We then brainstormed new ideas of how to provide vibrational feedback in a more toy-like system.”

The custom-designed plush toy houses sound transducers and a wireless communication system. The soft straps of the backpack/frontpack are adjustable, comfortable for the kids, and allow greater mobility for the dance workshop.

Natalie Neal, a junior mechanical engineering and materials science student, was inspired to create patterns and hand sew a series of plush toy monkeys that incorporate a Bluetooth receiver, audio amplifier, vibrational transducers and battery power supply. This new iteration, dubbed Vibrance, can be worn either as a backpack or a “frontpack” - what the team calls “hug mode.” Additional testing and user feedback led to supplementing the tactile feedback with a projected visualizer that produces colorful circles based on the audio input.

The Vibrance team presented their work at the Swanson School of Engineering’s fall 2018 Design Expo and swept the top three awards: first place in The Art of Making category, the People’s Choice Award, and the Best Overall Design. “Receiving those three awards really validated all of the hard work we did throughout the semester,” said Khan. 

The students’ innovative design has also received an enthusiastic response from kids, teachers, and parents. One parent of a child at WPSD wrote to the team, “I hope I’ll get the chance to see my son experience the vest vibration device. What an awesome idea!”

Dr. Samosky was recently awarded a Provost’s Personalized Education Grant to support high-potential – and potentially high-impact – student design projects like Vibrance, enabling them to continue beyond the class in which they originate and be nurtured toward real-world impact. The Vibrance team will continue to develop and improve Vibrance under this new Classroom to Community initiative in Dr. Samosky’s lab. The goal is to create a device that meets the needs of both WPSD and Attack Theatre, but most importantly, the team wants to continue to positively affect the lives of the children using their device. As stated by Jocelyn Dunlap, a senior communication science student, “We are heading back to WPSD to continue building a project that claims a spot in all of our hearts.”

###

This video of the Vibrance project, also created as part of the students’ coursework in The Art of Making, shows the system in action as it is used by instructors and kids at WPSD and with Attack Theatre. 

The Vibrance team includes, Issam Abushaban, a sophomore bioengineering and computer engineering student; Dani Broderick, a senior mechanical engineering student; Tom Driscoll, a junior computer engineering student; Jocelyn Dunlap, a senior communication science student; Austin Farwell, a junior mathematics student; Farah Khan, a senior bioengineering student; Stephanie Lachell, a senior mechanical engineering student; Evan Lawrence, a junior mechanical engineering student; Natalie Neal, a junior materials science and engineering student; Jesse Rosenfeld, a junior mechanical engineering student; and Caroline Westrick, a junior bioengineering student.