The future computer is a face

Interacting with computers has become so common that nobody seems to realize how big the gap between computers and humans is. Controlling computers requires many skills and an intuitive understanding of hardware characteristics, such as, the difference between temporary and permanent storage. The gap becomes evident when, especially older, novices interact with computers for the first time. Lacking the required knowledge or intuition they feel to perform even the simplest tasks. Despite the impressive progress in the speed and complexity of computer hardware and software during the past decades, there is little progress in reducing the gap between computers and humans.

The future human-computer interface will be similar to those of the omniscient computers featuring in TV shows such as Star Trek. The starship commander talks to the computer and the computer responds with a natural voice. In many science fiction shows, the fictitious computers are flat screens depicting a realistic human face. The virtual face makes eye contact with the human in front of the screen and smiles, frowns, or exhibits another appropriate facial expression. The future computer is likely to be such a virtual face. Current research in artificial intelligence is focusing on the development of software that enables computer interfaces to participate in natural dialogue with humans. A prominent example is the work of Don Bohus and co-workers at Microsoft Research. Bohus develops a system featuring an avatar face for human-like interaction with computer users. The major challenges are the realisation of virtual forms of verbal and nonverbal communication. Verbal communication requires speech recognition and speech generation. Automatic recognition of speech is a very hard problem, but the performance is improving albeit in very small steps. Also the generation of speech is challenging but gradually improving. Non-verbal communication requires the visual recognition and generation of facial expressions (and gestures). The automatic visual recognition of subtle facial expressions is a tough challenge. The same applies to the generation of human-like facial expressions, but here the developments are sped up by the movies and computer games industry. In animated movies such as Shrek (2001), the subtle facial expressions of the animated characters are directly derived from actors whose facial expressions are digitally sampled and copied onto the faces of animated characters.

At the Tilburg centre for Cognition and Communication, artificial intelligence experts and nonverbal communication experts teamed up to develop the next generation of software for recognising nonverbal expressions from video sequences. Master student Bart Joosten is applying and evaluating state-of-the-art facial-expression recognition methods to video sequences of children that have to solve easy and difficult mathematical puzzles. The challenge is to predict from their visual expressions if the children consider the puzzle easy or difficult. The preliminary results indicate that subtle facial expressions contain some information about how complex the puzzle is to the child. In the coming years TiCC researchers will continue to develop their face-reading software. The methods employed are based on algorithms that train the computer on recognising predictive visual features using, so-called machine learning algorithms. The availability at TiCC of an extensive database of videos of visual expressions under a wide variety of experimental settings, provides a valuable source for teaching the computer to infer the mental state of the user from his or her facial expressions. Maybe  in the near future, the virtual face on your wall-mounted computer screen welcomes you by asking you why you look so sad or happy.