My SCAD’s Journal

This project is to create a computer game which is interacted with the player’s facial expression. This game is designed not only for entertainment, but also for medical treatment such as facial physical therapy/speech therapy. In short, this project is to create a serious game called facial aerobics.

Thesis Description

In contemporary art, computer game is a type of interactive artwork. In the recent decades, lots of new generation of computer games have being developed, from the changes of game type to the transformation of participation experience. Due to the evolution of game type and interface design, the goal of computer game has been become not only for entertainment, but has additional purposes such as education and training, which is well-known as “Serious Game”.
Thanks to the electronic technology evolution, lots of new interfaces are developed. Human body is one of the new forms in this extension. A player uses his entire body (real or shadow), or part of his body (hands or head etc.) as an interface, to interact with the game world. Through this new experience of interaction, designers intent the players enjoy the game not only mentally, but also physically. In this design document, I would like to use a human face as an interface to navigate a game. Using existing face tracking software to capture a player’s facial expression, to transform the visual feedback from the reality into a meaningful command to the game world. As a result, this game can achieve multiple goals: entertainment and medical treatment purposes.

Facial Expression Recognition Toolkits

Below are some of the facial expression recognition toolkits available in the market:

In this project, I am going to use Attentive Interaction Design Toolkit as my interface mockup.

Interface Mockup using Attentive Interaction Design Toolkit

Facial Physical Therapy

Introduction

In psychology, facial expression is a kind of non-verbal communication. It reveals the innermost feelings of human beings. In 1972, Ekman et al. suggests six emotions can be detected in faces: happiness, surprise, anger, sadness, fear and disgust/contempt. 1
For the people who lose the ability to control their facial muscles, they lose their communication ability as well. In other words, facial paralysis is a disability of communication, both physiological and psychosocial. Clinically, facial paralysis has two types: Flaccid paralysis/paresis (complete facial paralysis), and Synkinesis/mass action (varying degrees of motor weakness accompany by abnormal, asynchronous facial movements). 2 Different type of facial paralysis requires different treatment tactics.

Treatment – Neuromuscular Retraining

Nowadays, physical therapy is still the recommended treatment for facial paralysis. “Gross facial exercises, massage, electrical stimulation, and prosthetic devices or taping to lift a drooping, flaccid face” 2 are the traditional therapy techniques in decades ago. However, most therapists found that they are ineffective. Thirty years ago, neuromuscular retraining (NMR) technique appeared in the literature, now known as an effective way for optimal recovery from facial nerve paresis. 2
This technique is “a marriage of neurophysiology, psychology, therapeutic science, learning theory, and art.” 2 It is using surface electromyography (EMG) feedback and specific mirror exercises to modify the manner in which patients contracted their muscles. However, there are no generic lists of exercises for the patients because clinicians found that each patient has his own functional profile. 2

Facial Movements

Clinically, facial movement falls into two types: volitional and spontaneous. In some cases, a patient lost the ability of expressing emotions but still have the ability of volitional control (Figure 1, right column). Contrarily, there is also a case that a patient is able to produce emotional expression but failed to move their facial muscle voluntarily (Figure 1, left column). 3

Figure 1. Voluntary and emotional facial paresis 3

In this case, my game is more suitable for the patient who has the ability of volitional control.

Volitional Movement vs. Facial Muscle Groups

Figure 2. Major facial muscle groups 4

Table 1. Volitional movement vs. facial muscle groups 2 , dotted lines indicates the expressions can be recognized easily.

From this table, we have the idea of what expression movements are exercising which facial muscle groups.

Example Clinical Program For Facial Physical Therapy

Figure 3. Personal exercise program from Chatham Sports Medicine & Physical Therapy

Speech Therapy

Introduction

Speaking is our most significant channel of communication. Orofacial Myofunctional Disorders are a type of speech disorders/communication disorders where ‘normal’ speech is disrupted. “Orofacial Myofunctional Disorders (OMD) involve behaviors and patterns created by inappropriate muscle functional and incorrect habits involving the tongue, lips, jaw and face.” 6 The excessive pressure caused by OMD may result in speech problem. Tongue thrust and abnormal lips resting position are the most common orofacial myofunctional variations. 6

Treatment – Oral Facial Myofunctional Therapy

Oral facial myofunctional therapy is the common speech therapy program for ODM. It provides therapeutic oral and facial exercises to develop appropriate tongue and lip resting postures. “Oral Aerobics” 7 is one of the existing treatment programs for ODM. Oral Aerobics is a videotape program which provides a series of lip and tongue exercises together with dynamic music. Below table is a list of lip and tongue exercises provided by the videotape:

The preview video clip can be downloaded from http://www.trainyourbrainco.com/paerobics.htm. This video is one of my major references for the facial expression patterns using in my game.