Suche

Modeling Multimodal Integration with Event Logic Charts

Gregor Mehlmann and Elisabeth André

Modeling Multimodal Integration with Event Logic Charts

erschienen 2012 Proceedings of the 12th International Conference on Multimodal Interaction, ICMI' 12, pp. 125-132, Santa Monica, CA, USA, October 22-26, 2012


In this paper we present a novel approach to the combined modeling of multimodal fusion and interaction management. The approach is based on a declarative multimodal event logic that allows the integration of inputs distributed over multiple modalities in accordance to spatial, temporal and semantic constraints. In conjunction with a visual state chart language, our approach supports the incremental parsing and fusion of inputs and a tight coupling with interaction management. The incremental and parallel parsing approach allows us to cope with concurrent continuous and discrete interactions and fusion on different levels of abstraction. The high-level visual and declarative modeling methods support rapid prototyping and iterative development of multimodal systems.