Howard Rose
Voice/Gesture Recognition
Mark Billinghurst
Additional Assistance
Ari Hollander, Paul Schwartz and Jesus Savage
|
|
|
|
Howard Rose Voice/Gesture Recognition Mark Billinghurst Additional Assistance Ari Hollander, Paul Schwartz and Jesus Savage |
Watch a short video introduction of Zengo Sayu
Quicktime MPEG4
Download WMV
Zengo Sayu is a virtual environment where students learn Japanese through direct experience with the language. The environment is populated with objects that talk and help students learn vocabulary, sentence structures and grammar through meaningful, self-directed activities. Activities increase in complexity leading to a construction game where students follow and give commands to arrange building blocks. The game can be played either by a single student with the computer, or by groups of students interacting within the same virtual space.This virtual reality approach for teaching foreign language enables students to take control of their own learning and explore the language in an environment built to support them. Zengo Sayu is a direct method of language instruction which conveys meaning directly through readily understood objects and situations rather than relying on English translations.
Zengo Sayu uses digitized voice samples for natural speech reproduction. The environment also includes voice and gesture recognition capabilities which greatly expand the useful possibilities for this educational approach. Imprint is in the process of developing a robust application for use on a desktop PC.
**The complete Zengo Sayu system includes voice and gesture recognition technologies and was developted at the Human Interface Technology Laboratory by Howard Rose.
**Real-time rendering with Division's dVS and dVISE running on an SGI Onyx.Zengo Sayu's four learning modes.
1. Five Colors and One Noun: Students establish builds basic vocabulary through experience and listening. This mode demonstrates the most rudimentary use of the environment as a talking dictionary.
2. Preposition Table: Students learn a new noun (table), 5 prepositions and begin to hear words used in longer phrases. Mode Two activities require students to perceive and act in all three dimensions of the virtual environment.
3. Free Experimentation, Using the Query Wand: The student continues to learn and practice by building structures. Learning is tested and supported by aspects of the environment such as the Query Wand, which allows the student to pose questions about specific object relationships.
4. Following Commands, Building Models: Mode Four tests students' progress in the context of a construction game. At the simplest level, the student is shown a model which she must recreate while listening to verbal commands. The game becomes progressively more challenging when the model is hidden from the student, forcing her to rely only on the verbal commands. Zengo Sayu can also become a multi-player game, where two or more students learn and test each other within the same virtual environment.
Further Reading on Zengo Sayu:
*All rights to the documents listed below are held and maintained by the author or Imprint Interactive Technology Inc. unless otherwise noted.
(1996) Zengo Sayu: An Immersive Educational Environment for Learning Japanese: Final Report to The Washington Technology Center Available as RTF Document
(1996) Design and Construction of a Virtual Environment for Japanese Language Instruction. Available as HTML Document (Masters Thesis)
(1995) Rose, Howard and Billinghurst, M. Zengo Sayu: An Immersive Educational Environment for Learning Japanese.HITL Technical Report No. TR-95-4. HTML Document
