Multimodal Interfaces for Internet

Abstract

The World Wide Web frightened us: during the first two years of its popularity, we felt that human-computer interaction had been set back 30 years. Fortunately, Java expanded the possibilities for user interface design, providing a way to run complex programs over the net. In
this paper, we present a Java-enabled application with a multimodal (pen and voice) interface over the web.

Our implementation approach was to add Java to the set of languages accepted by the Open Agent Architecture (OAA), a framework for rapidly prototyping complex applications, and particularly suited to those with multimodal interfaces [1]. Given the OAA’s distributed nature, an OAA-based application can be run from a lightweight computer by downloading only the small user interface component, while the core of the application is implemented on a server composed of larger agents (e.g. speech recognition (SR), natural language, database) which cooperate and compete in parallel.

Despite the current lack of APIs for media input in Java, we chose voice entry as the primary input modality for the user [2], first by using a telephone to access a remote Nuance Speech Recognition server [3], and secondly by designing our own Java API for Speech Recognition using external native methods on the client.

ATIS [4], our first prototype application using SR over the telephone and the Java implementation of the OAA, has been publicly available on our web site for more than a year. Given the success of this first experiment (more than 4000 users), our next task was to bring multimodal concepts developed for several map-based applications [5] to the Web. In order to achieve our multimodal objectives, it was necessary to adapt the pen modalities (gestures and handwriting) to an Internet and multiplatform context, i.e. the pen had to be interchangeable with any pointing device. For portability, the gesture recognition algorithms [6] were recoded directly in Java. A trade-off was made for handwriting, integrating a Java-enabled version of JOT, a character by character recognizer developed by CIC [7].