RAVE Revolutionary Learning Tool Prototype*

 NEW! Click here to see RAVE featured in Newsweek (October 15, 2017)

 

Dr. Laura-Ann Petitto (Keck and NSF-INSPIRE Principal Investigator) with baby interacting with RAVE learning tool prototype.

Prof. Laura-Ann Petitto (Principal Investigator, Keck and NSF-INSPIRE grants) and baby interacting with RAVE learning tool prototype.

There are a vast number of children throughout the nation and world who suffer from the devastating effects of minimal or delayed language exposure. This can impact children in various home environments with reduced adult contact, particularly socially rich language interactions. This is especially challenging for over 90% of young deaf infants who initially have little (or no) exposure to a fully accessible natural signed language. Minimal or delayed early first-language exposure can have a lifelong negative impact on healthy learning, cognitive development, language acquisition, and reading success.

Professor Laura-Ann Petitto (Principal Investigator, grants from the W. M. Keck Foundation and NSF INSPIRE, IIS-1547178), the Petitto Brain and Language Lab for Neuroimaging (BL2) at Gallaudet University, additional scientists from three universities, and spanning four disciplines, aim to solve this problem with a revolutionary learning tool called the Robot AVatar thermal-Enhanced prototype, or RAVE.

What is RAVE?
Created and led by Petitto with funding from two (2) three-year research grants from the W.M. Keck Foundation and the National Science Foundation (NSF-INSPIRE), RAVE is a robot-avatar learning tool that senses when babies’ emotional-attentional interest is most engaged and “ready to learn” (with Thermal Infrared imaging and eye-tracking). If so, RAVE’s thermal system then triggers the robot and virtual human to start a variety of social interactions with the baby, and especially triggers the virtual human when to begin, and when to stop, language conversations and nursery rhymes with the baby in American Sign Language. (A speech option is planned for later.)

The work grows out of Petitto’s brain imaging discoveries using functional Near Infrared Spectroscopy (fNIRS) showing that all infants are sensitive to highly specific patterns at the heart of human language structure in early life: specifically, rhythmic temporal patterns with maximal contrast about the size of phonetic-syllabic units in natural language—all natural language, be it signed language or spoken language (see Petitto publications; Petitto et al., 2016). Like a lock and key, RAVE is designed to provide babies with minimal language exposure with just those specific language patterns that they are most sensitive to in early life and just when their developing brains need to experience it most to ensure latter language and reading success.

Depending on the babies’ communicative interactions, RAVE’s virtual human will produce rhythmic-temporal nursery rhymes specifically designed to help all babies discover the words, small meaningless parts of words (phonetic-syllabic units) and their patterning to promote healthy language growth.

One very special feature of RAVE is that it can change the types of language interactions it has with the baby based on the baby’s own communications. Here, the virtual human is built to produce language communications that are meaningfully tied to what the baby is doing and communicating. For example, if a baby points to the virtual human, the virtual human may respond, “Are you pointing to me?” and may then begin to sign a happy nursery rhyme in ASL. Thus, beyond being socially interactive, RAVE makes possible first-time human-machine interactions that approximate a meaningful socially contingent conversation (see Petitto Refereed Abstracts; Petitto, 2017).

These scientific achievements permit for the first time the advancement of a machine-human device that can (1) engage in socially contingent rudimentary conversations with young babies (ages 6-12 months), (2) provide natural language when babies are most “ready to learn” (even before they can even produce language to communicate this to us), and (3) provide just the right type of honed language stimulation (with specific rhythmic frequencies) that their brains require to “wedge” or “hold open” neural systems for language learning until fuller, richer language input becomes accessible.

The leading outcome goal of RAVE is to produce an augmentative learning tool that makes available natural language patterns to young babies during the precise critical or sensitive periods in early brain and behavior development when they need it most.

To build RAVE, basic science studies were first conducted with Dr. Petitto’s fNIRS brain imaging system that measures language and higher cognitive processing in the infant brain with Dr. Merla’s Thermal IR imaging system that measures baby’s levels of emotional-attentional interest. This work yielded evidence that (a) all babies’ are sensitive to specific phonetic-syllabic rhythmic patterning at the heart of human language structure (see Petitto references, e.g., Petitto et al., 2016) and that (b) increased interest to these specific aspects of human language patterns are coupled with peaked emotional arousal as determined by the Thermal IR imaging (Merla/Universita Gabriele D’Annunzio, Chieti-Pescara, Italy). Following this basic research, Dr. Merla and team built algorithms to signal a baby’s emotional state to the avatar, specifically, to stop and start its language productions (Avatar science: Traum team/USC), which was further interfaced with the robot’s solicitation actions to the baby (e.g., head nodding, eye blinking, etc.; Robotics science: Scassallati team/Yale).

See Petitto and team’s experiments at the heart of RAVE in action. VIDEO WATCH: Petitto BL2 – What is it like for Babies to Participate in Our Brain Imaging Studies? 

How RAVE Works
The RAVE learning tool
is to be placed near a baby’s crib or seat to facilitate a sensitivity to language patterns vital for language and later reading success (see Figure 1).

When a baby in a heightened state of cognitive and social-emotional interest (“ready to learn”), the robot and/or the avatar are triggered to start, stop, or solicit interaction based on the baby’s social-emotional engagement. The baby’s interactive eye gaze with either the robot or the avatar then triggers the avatar to produce rhythmic-temporal nursery rhymes, fairy tales, or simple social conversations in a visual (sign) language, eventually with speech options. The avatar either proceeds or stops socially-contingent conversational interactions with the baby depending on the baby’s level of social engagement.

Dr. Barbara Manini (Post-Doc Petitto Lab/Thermal IR Imaging, Gallaudet University) interacting with RAVE robot while baby observes.

Technological Advances in RAVE
What is remarkable is that RAVE — with two non-human, artificial agents (robot and avatar) — once completed, will be able to simulate
for the first time a conversation with a baby in ways that are like a human mother and her baby’s jointly shared social attention, and with basic avatar conversations that are meaningful and socially contingent on one another.

Petitto’s multidisciplinary team recently launched a pilot of RAVE to test its effectiveness. Learn more about the RAVE pilot week. Results of the pilot will be shared in late fall 2017.

RAVE is a revolutionary learning tool that sheds light on how all babies discover core components of language. This knowledge can translate into early learning gains for populations that would otherwise be at a lifelong disadvantage.

RAVE also provides the nation with a competitive science and technological edge.

This work further addresses the need for underrepresented groups to participate in science, technology, engineering and mathematics; students in interdisciplinary science; and young deaf scientists participate in the advancement of science with broad implications for society.

Collaborative efforts to address minimal language exposure will enable the RAVE team to make advancements in four sciences: Developmental Cognitive Neuroscience, Virtual Human Science, Robotics and Applied Psychophysiology/Biomedical Engineering.

Many children suffer from minimal language exposure for a variety of reasons. Some groups can be beyond “minimal” language exposure to having no accessible language exposure in early life—especially young deaf children in non-signing homes. RAVE will provide nursery rhymes and socially-rich language conversations to young babies in American Sign Language during the precise sensitive periods of early language acquisition when it is needed most for healthy early brain, language, and cognitive growth.

RAVE scientists expect it to be an augmentative learning tool, never to replace human interactions, but to assist early language learning. The hope is that one day RAVE will enhance language interactions with young babies who suffer from the harmful effects of minimal language exposure.

*The RAVE learning tool is a prototype and currently is being tested with hearing and deaf infants. RAVE is presently not in production or available for purchase by the general public.

Prof. Laura-Ann Petitto (Principal Investigator, Keck and NSF-INSPIRE grants) with a Gallaudet University undergraduate student, TraciAnn Hoglind, during early experiments with RAVE’s MAKI robot.

Contact Us
If you would like your infant to be considered for the RAVE project, please contact bl2@gallaudet.edu.  

About the RAVE Team
The RAVE team explores a technological innovation for children with minimal or no language input. The Petitto Brain and Language Lab for Neuroimaging (BL2) conducted the functional near infrared spectroscopy (fNIRS) brain imaging studies of infants’ sensitivity to specific rhythmic temporal patterns in language and at what age in human development, and provides RAVE’s eye-tracking technology. Petitto and her BL2 team work alongside a multidisciplinary team:  At Gallaudet, Melissa Malzkuhn collaborates with Dr. Petitto’s BL2 through Malzkuhn’s creation of the ASL nursery rhyme avatars (in the NSF-Gallaudet Science of Learning Center, VL2’s Motion Light Lab with its advanced Motion Capture Studio; Malzkuhn Creative Director); Dr. David Traum from the University of Southern California (Virtual Human/Avatar Science), Dr. Brian Scassellati from Yale University (Robotics), and Dr. Arcangelo Merla from the Università Gabriele D’Annunzio in Chieti-Pescara, Italy (Applied Psychophysiology who pioneers both the Thermal Infrared Imaging and the Face Imaging and tracking in RAVE).

Pictured from left to right: Setareh Nasihati Gilani (Graduate Student, Traum’s Virtual Human Lab, University of Southern California); Prof. Arcangelo Merla (Thermal IR Imaging Scientist, Università Gabriele D’Annunzio, Chieti-Pescara, Italy); Dr. Barbara Manini (Post-Doc Petitto Lab/Thermal IR Imaging, Gallaudet University); Prof. Brian Scassellati (Robotics Scientist, Yale University); Prof. Laura-Ann Petitto, P.I. (Cognitive Neuroscientist, fNIRS brain imaging, Gallaudet University); Prof. David Traum (Virtual Human Scientist, University of Southern California); Dr. Katherine Tsui (Post-Doc, Scassellati Robotics Lab, Yale University); Melissa Malzkuhn (Creative Director ML2 and Mocap Studio, Gallaudet University); Jason Lamberton (Mocap Scholar, ML2 and Mocap Studio, Gallaudet University)
Not pictured: Jake Noah Brawer (Graduate Student Scassallati’s lab, Yale University); Geo Kartheiser (Ph.D. in Educational Neuroscience program Graduate Student, Petitto Advisor, Gallaudet University); Dr. Anton Leuski (Virtual Human Scientist, University of Southern California); Dr. Ari Shapiro (Virtual Human Scientist, University of Southern California); Adam Stone {Ph.D. in Educational Neuroscience Graduate Student, Petitto Advisor, Gallaudet University (now Dr. Adam Stone)



About Gallaudet
Gallaudet University, federally chartered in 1864, is a bilingual, diverse, multicultural institution of higher education that ensures the intellectual and professional advancement of deaf and hard of hearing individuals through American Sign Language and English. Gallaudet maintains a proud tradition of research and scholarly activity and prepares its graduates for career opportunities in a highly competitive, technological, and rapidly changing world. Learn more.  

About the National Science Foundation
The National Science Foundation (NSF) is an independent federal agency that supports fundamental research and education across all fields of science and engineering. In fiscal year (FY) 2017, its budget is $7.5 billion. NSF funds reach all 50 states through grants to nearly 2,000 colleges, universities and other institutions. Each year, NSF receives more than 48,000 competitive proposals for funding and makes about 12,000 new funding awards. For more information, please visit https://www.nsf.gov.

About the W.M. Keck Foundation
Based in Los Angeles, the W. M. Keck Foundation was established in 1954 by the late W. M. Keck, founder of the Superior Oil Company. The Foundation’s grantmaking is focused primarily on pioneering efforts in the areas of medical, science and engineering research. The Foundation also maintains an undergraduate education program that promotes distinctive learning and research experiences for students in the sciences and in the liberal arts, and a Southern California Grant Program that provides support for the Los Angeles community, with a special emphasis on children and youth from low-income families, special needs populations and safety-net services. For more information, please visit www.wmkeck.org.