KUBILINGO

KubiLingo is a conversational robot that carries out foreign language lessons autonomously or with partial human supervision, for individuals and small groups.

This project was conducted at the Human-Centered Robotics Lab at UW, in partnership with a non-profit Morroccan foundation that builds schools across Africa.

The Team

We were an interdisciplinary team of researchers, designers and engineers from Computer Science, HCDE and Department of Statistics, mentored by Prof. Maya Cakmak, who heads the Human Centered Robotics Lab at UW CSE.

Leah Perlmutter
Sahil Anand
Alex Fiannaca

Lindsey Arnold
Eric Kernfeld
Maya Cakmak

My Role

I was the Robot Experience Designer in our team. My responsibilites included designing the interactions between the robot and the user. I created sketches, wireframes, mockups, visual assets, animations and prototyped the robot behaviors. I also moderated the studies with participants, and co-authored a conference paper which was accepted at a major robotics conference.

THE PROBLEM

English is one of the most common languages in the world and it is arguably the most useful language to know. Speaking English means access to abundant information resources, including a large majority of all online content, published scientific papers, and many other educational resources.

NRRI Retirement Risk Index
Top 10 Languages on the Internet
NRRI Retirement Risk Index
Most common second language (L2) in African Countries

In many countries English is the main second language (L2) taught in public schools as early as in kindergarten, particularly in countries that have been colonized by the British Empire in the last century. However, in some developing countries, English has not widely spread as part of public education curricula. For instance, most African countries have typically taught French as L2.

This has limited the flux of students from these countries to English-speaking international academic institutions and isolated African institutes from the larger research community. The schools that wish to offer English as a second language in these countries, often have difficulty finding qualified English teachers due to this historical bias and have difficulty attracting foreign teachers due to political instability or limited resources.

WHY USE A ROBOT?

Our motivation stems from the results of recent studies demonstrating the effectiveness of robotic instruction in comparison to other electronic media and also from the recent increase in the availability of reliable, low-cost robotic platforms.

Storytelling robot by MIT
Nao Robot by Aldebaran Robotics

Researchers have demonstrated that the physical presence of an embodied robot results in greater learning gains than when the same content is presented without a robot, using simple computerized lessons. As robotics researchers, we were interested in exploring the possibility of using low-cost robotic agents for education in developing countries as well as understanding the challenges in designing social and conversational robots.

Conference Paper

“Automatic Adaptation of Online Language Lessons for Robot Tutoring”

— Leah Perlmutter, Alexander Fiannaca, Eric Kernfeld, Sahil Anand, Lindsey Arnold, and Maya Cakmak


In August 2016, our paper on Conversational Robotic Tutors was accepted at ICSR 2016, International Conference on Social Robots.

Design Journey

Designing an experience for a robot with a screen, a moving base, and voice interactions together was something totally new to me. My goal was to make the interactions natural, personal, fun and engaging.

Sketches and Wireframes

I sketched a number of concepts for the elements on the screen (the face) of the robot. The screen served as a canvas for drawing the eyes of the robot, as well as displaying the question prompts.

Overall Architecture
Interface Layout
Overall Architecture
Question Type 1
Overall Architecture
Question Type 2
Overall Architecture
Question Type 3

These low-fidelity sketches helped me communicate my ideas effectively with my team and also receive honest and valuable feedback on my designs.

Mockups and Interaction Map

I then created mockups and interaction maps of the refined concepts after incorporating some of the feedback.

The Experiential Telepresence System
Interaction Map

This exercise forced me to think more thoroughly about all the elements on the page from top to bottom, not just the basics, and how the interactions will look like.

Design Inspiration

I chose a vibrant and balanced color scheme similar to Duolingo's to make the learning experience fun and playful. I tried different color palettes for the UI and asked everyone on the team for their feedback.

Overall Architecture
Different color palettes for Kubi

Kubi was finally designed to resemble Duo, the owl mascot of Duolingo. Kubi's face is green like Duo's feathers, and it's hands are orange, like Duo's feet and beak.

Overall Architecture
Duo and Kubi

Adapting Lessons from Duolingo

We created a mapping plug-in for chrome to automatically convert Duolingo's lessons into robot lessons. In Duolingo, the basic unit of interaction is a prompt, in which some material (text, audio, pictures) is delivered to the user and the user is expected to respond by typing or clicking. We deconstructed this prompt into smaller conceptual parts like directive, body, hints, feedback, progress, etc. A detailed description of these directives is present in the conference paper.

Overall Architecture
The chronological progression of a prompt on Duolingo (top) and KubiLingo (bottom). User actions are described in the gray arrows.

We implemented the mapping for five different types of prompts. These five prompts account for all the material taught in the lessons we used for evaluating our system.

Animations

“When you want to arouse emotions, it doesn’t matter so much how something looks, it’s all in the motion, in the timing, and how the thing moves.”

— Guy Hoffman

My aim was to make Kubi appear emotionally expressive and life-like, just like the cartoon characters from the famous Disney animations. All the animations for Kubi's facial expressions follow the guidelines of the "Principles of Animation", introduced by the Disney animators Ollie Johnston and Frank Thomas in their 1981 book "The Illusion of Life".

Overall Architecture
Look Around
Overall Architecture
Happy Expression
Overall Architecture
Sad Expression
Overall Architecture
Shocked Expression
Overall Architecture
Blink Twice
Overall Architecture
Curious Expression

Secondary Action

The facial animations are also accompanied by secondary actions in the form of neck movements. Secondary actions added life and naturalness to the character by supporting the main action.

Idle Motions

Kubi also has idle motions in the form of random head movements and periodic eye blinking. Previous research on robots portraying idle motions has shown participants to perceive the robot as more alive and empathic compared to robots without idle motions.

Putting it together

Here's a demo of an interaction between Kubi and a participant from our study. We selected Dutch and Swedish languages for Kubi to teach the participants as it was easy to find participants having no prior knowledge of these languages.

Study Results

We tested our system in a user study to compare it's performance with Duolingo, the source of the lesson content. Our results did not conclusively show that either KubiLingo or Duolingo has greater learning gains, which means we haven't made the lessons detectably worse teaching them with KubiLingo. KubiLingo's effectiveness in on par with Duolingo's, however, there is much room for improvement in the usability of our system which should increase it's likeability, reduce distraction, and lead to learning gains, potentially exceeding Duolingo's effectiveness.

The Experiential Telepresence System
Summary of the most common user feedback. Number of users in parentheses.

Users found KubiLingo to be more fun and entertaining, but when it comes to preference, user reported that the screen felt more natural and that they would prefer to use the screen to learn a language in future. A detailed analysis of the results is presented in the conference paper.

Lessons Learnt

I really enjoyed collaborating on this project. We had a fantastic team working on something we all were passionate about - ROBOTS! Some of my main takeaways were

  • Motion and animation can bring personality and liveliness to a character. Programming subtle idle motions into the robot made the robot feel alive.
  • Variation in behavior and personalization are important qualities of social robots. Our robot was designed with a fixed set of behaviors and animations. Once a user learnt about all these behaviors, he/she no longer showed the same level of excitement as in the beginning of the interaction.
  • Novelty factor has a huge effect on testing such new technologies with users. Users may initially find the robot fun and engaging, but it does not mean that they will continue to find it fun and engaging after an extended period of time. Longitudinal studies can help us uncover such insights.