Carnegie Mellon University

Image of app team

September 18, 2018

Education App Uses Photos To Help People Form Proper Sentences

CMU Design for America project assists people with language deficits

By Byron Spice

Byron Spice
  • School of Computer Science
  • 412-268-9068

Students in the Carnegie Mellon University chapter of Design for America (DFA) have developed a free mobile app that uses visual cues to help children and adults — particularly those with language difficulties — form meaningful sentences that are grammatically correct.

The education app, called Sentence Mosaics, uses photos and color-coded parts of speech to prompt users as they construct sentences.

The app is useful for elementary students and for a wide range of children and adults who are on the autism spectrum, or who have developmental disabilities, hearing disorders, stroke-related language deficits, dementia and other problems with language, said Alyse Baker, a certified speech-language pathologist who has honed the system for decades.

"Alyse has used colored blocks, colored cards and foam tiles attached with Velcro to a foam board as aids, but that's not practical for use outside the classroom," said Mehar Sawhney, a DFA@CMU member and a senior majoring in mathematics and physics. "We worked with her to translate her system into an iPad app that can be taken anywhere."

Baker developed Sentence Mosaics based on longstanding research and practice within the field of speech-language pathology. She has long recognized that transforming the system into an app would make sense, but found the prospect daunting.

"When I first looked into developing an app, I was told it would cost $20,000 to $30,000 to write the code," Baker said. "I did not have that much money in my hip pocket, so I knocked on doors for six years until a stroke of luck took me to David Kosbie."

Kosbie, an associate teaching professor of computer science at CMU, connected her two years ago with DFA@CMU, an interdisciplinary group of students who design and engineer solutions for social good.

A group of DFA members, including Sawhney, computer science majors Elizabeth Board and Sashank Gogula, and cognitive science major Elizabeth La, worked with Baker to translate physical aids into digital ones. Jennifer Chou, who graduated last spring with a computer science degree, also was a major contributor to the project.

Photos are key to the system. Users write sentences that describe actions depicted in photos, preferably photos that include the student or show a scene from their everyday life. Board said the app can import photos from an iPad or iPhone.

Baker explained that the photos make the subject real to students, which for many is a necessary step before they can write.

"Some of my students, if they didn't have a dog of their own, literally could not write a sentence about a dog," she added.

The app is designed to be used with a teacher, parent or other "coach," who helps by loading photos into the app and by designating a string of colored boxes that signal to the student which parts of speech — nouns, pronouns, verbs, adverbs, adjectives, etc. — are necessary to describe the photo.

"Without the intercession of Dave Kosbie and countless hours of programming by students at Carnegie Mellon, this app would not have become a reality," Baker said.

More information about the app is available at https://sentence-mosaics.herokuapp.com/. The free iOS app can be downloaded from the App Store.