Skip to main content
Log in

A third modality of natural language?

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

In the late Eighties the Natural Language Processing community began appreciating the role of multimodality in interactive systems. Intelligent multimodal systems are systems that integrate natural language (generally so far keyboard-based input, shortly also voice) with other media such as gestures in input or graphics in output. The perspective of what can be called visible interactive communication is discussed and considered as a possible new modality of natural language, after the spoken and the written ones. This should not be confused with the type of hypermedia that are now being developed. There, basically, the interface space is finite, even if one dimension may be added. Here the infinite creativity of human language is potentially preserved as the fundamental communication instrument.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Allgayer, J., Harbusch, K., Kobsa, A., Reddig, C., Reithinger, N. & Schmauks, D. (1989). XTRA: A Natural-Language Access System to Expert Systems.Intl. J. on Man-Machine Studies 31: 161–195.

    Google Scholar 

  • Appelt, D. E. (1985).Planning English Sentences. Cambridge University Press.

  • Arens, Y., Miller, L., Shapiro, S. C. & Sondheimer, N. K. (1988). Automatic Construction of User Interface Displays. In Proceedings ofThe Seventh AAAI Conference. St. Paul, Minnesota.

    Google Scholar 

  • Arens, Y. & Hovy, E. (1990). How to Describe What? Towards a Theory of Modality Utilization. In Proceedings ofThe Twelfth Cognitive Science Conference. Cambridge, MA.

  • Barrett, E. (1989). Textual Intervention, Collaboration, and the Online Environment. In Barrett, E. (ed.)The Society of Text. MIT Press.

  • Bush, V. (1945). As We May Think.Atlantic Monthly 7: 101–108.

    Google Scholar 

  • Carenini, G., Pianesi, F., Ponzi, M. & Stock, O. (1993). Natural Language Generation and Hypertext Access.Applied Artificial Intelligence 7: 135–164.

    Google Scholar 

  • Cohen, P. R., Dalrymple, M., Moran, D. B., Pereira, F. C. N., Sullivan, J. W., Gargan Jr, R. A., Schlossberg, J. L. & Tyler, S. W. (1989). Synergetic Use of Direct Manipulation and Natural Language. In Proceedings ofThe CHI'89, 227–233. Austin, Texas.

  • Conklin, J. (1987). Hypertext: An Introduction and Survey.IEEE Computer 20(9).

  • Dahlbäck, N. & Jönsson, A. (1989). Empirical Studies of Discourse Representations for Natural Language Interfaces. In Proceedings ofThe Fourth Conf. European Chapter of the ACL, 291–307. Association for Computational Linguistics.

  • Dale, R. (1992). Visible Language: Multimodal Constraints in Information Presentation. In Dale, R., Hovy, E., Rösner, D. & Stock, O. (eds.)Aspects of Automated Language Generation, 281–283. Springer.

  • Feiner, S. & McKeown, K. (1990). Coordinating Text and Graphics in Explanation Generation. In Proceedings ofThe AAAI-90, 442–449. Boston, MA.

  • Franconi, E. (1990). The YAK Manual: Yet Another KRAPFEN.IRST Manual 9003-01. IRST. Trento, Italy.

  • Grosz, B. J. & Sidner, C. L. (1986). Attention, Intentions and the Structure of Discourse.Computational Linguistics 12(3): 175–204.

    Google Scholar 

  • Halasz, F. G. (1988). Reflections on NoteCards, Seven Issues for the Next Generation of Hypermedia Systems.Communications of the ACM 31(7).

  • Hollan, J., Rich, E., Hill, W., Wroblenski, D., Wilker, W., Wittenburg, K. & Grudin, J. (1988). An Introduction to Hits: Human Interface Tool Suite.MCC, Tech. Rep. ACA-HI-406-88. Austin, Texas.

  • Hovy, E. (1988).Generating Natural Language under Pragmatic Constraints. Lawrence Erlbaum Associates.

  • Hovy, E., & Arens, Y. (1991). Automatic Generation of Formatted Text. In Proceedings ofThe Ninth AAAI Conference, 92–97. Anaheim, CA.

  • Kass, R. & Finin, T. (1988). Modeling the User in Natural Language Systems.Computational Linguistics 14(3): 5–22.

    Google Scholar 

  • Kobsa, A. & Wahlster, W. (eds.) (1989).User Models in Dialog Systems. Springer.

  • Kurohashi, S., Nagao, M., Sato, S. & Murakami, M. (1992). A Method of Automatic Hypertext Construction from an Encyclopedic Dictionary of a Specific Field. In Proceedings ofThe Third Conference on Applied Natural Language Processing, 239–240. Trento, Italy: Association for Computational Linguistics.

    Google Scholar 

  • Lay, K. Y., Malone, T. W. & Yu, K. C. (1988). Object Lens: A “Spreadsheet” for Cooperative Work.ACM Transaction on Office Information Systems 6: 332–353.

    Google Scholar 

  • Lavelli, A., Magnini, B. & Strapparava, C. (1992). An Approach to Multilevel Semantics for Applied Systems. In Proceedings ofThe Third Conference on Applied Natural Language Processing, 17–24. Trento, Italy: Association for Computational Linguistics.

    Google Scholar 

  • Lavelli, A. & Stock, O. (1990). When Something is Missing: Ellipsis, Coordination and the Chart. In Proceedings ofThe Thirteenth International Conference on Computational Linguistics, COLING-90, 184–189. Helsinki, Finland: Association for Computational Linguistics.

    Google Scholar 

  • Lemke, A. & Fischer, G. A. (1990). Cooperative Problem Solving System for User Interface Design. In Proceedings ofThe Eighth AAAI Conference, 479–484. Boston, MA.

  • MacLaughlin, D. M. & Shaked, V. (1989). Natural Language Text Generation in Semi-Automated Forces.BBN Report 7092.

  • Mackinlay, J. D. (1986).Automatic Design of Graphical Presentations. Ph.D. dissertation, Stanford University.

  • Maybury, M. T. (1991). Planning Multimedia Explanations Using Communicative Acts. In Proceedings ofThe Ninth AAAI Conference, 61–66. Anaheim, CA.

  • Maybury, M. T. (ed.) (1993).Intelligent Multimedia Interfaces. AAAI Press/MIT Press: Menlo Park, CA/Cambridge, MA.

    Google Scholar 

  • Moore, J. D. & Swartout, W. (1990). Pointing: A way toward explanation dialogue. In Proceedings ofThe Eighth AAAI Conference, 457–464. Boston, MA.

  • Nelson, T. H. (1981).Literary Machines. Swarthmore, PA 19801: Nelson, T., P.O. Box 128.

  • Oviatt, S. L. & Cohen, P. R. (1989). The Effects of Interaction on Spoken Discourse. In Proceedings ofThe Twenty-Seventh Meeting of the ACL, 126–134. Vancouver, Canada: Association for Computational Linguistics.

    Google Scholar 

  • Oviatt, S. L. & Cohen, P. R. (1991). discourse Structure and Performances Efficiency in Interactive and Noninteractive Spoken Modalities.Computer Speech and Language 5(4): 297–326.

    Google Scholar 

  • Paris, C. L. (1987). Combining discourse Strategies to Generate Descriptions to Users Along a Naive/Expert Spectrum. In Proceedings ofThe IJCAI-87, 626–632. Milan, Italy.

  • Pianesi, F. (1993). Head Driven Bottom Up Generation and Government and Binding: A Unified Perspective. In Horacek, H. & Zock, M. (eds.)New Concepts in Natural Language Generation, 187–214. Printer: London.

    Google Scholar 

  • Reiter, E., Mellish, C. & Levine, J. (1992). Automatic Generation of On-Line Documentation in the IDAS Project. In Proceedings ofThe Third Conference on Applied natural Language Processing, 64–71. Trento, Italy: Association for Computational Linguistics.

    Google Scholar 

  • Samek-Lodovici, V. & Strapparava, C. (1990). Identifying Noun Phrase References, the Topic Module of the ALFRESCO System. In Proceedings ofThe ECAI-90, 573–578. Stockholm, Sweden.

  • Schmauks, D. (1987). Natural and Simulated Pointing. In Proceedings ofThe Third Conference of the European Chapter of the ACL, 179–183. Association for Computational Linguistics.

  • Slack, J. & Conati, C. (to appear). Modeling Interest: Exploration of an Information Space. To appear inActa Phsychologica on Cognitive Ergonomics.

  • Stefik, M., Foster, G., Bobrow, D., Kahn, K., Lanning, S. & Suchman, L. (1987). Beyond the chalkboard: Using computers to support collaboration and problem solving in meetings.Communication of the ACM 30(1): 32–47.

    Google Scholar 

  • Stock, O. (1989). Parsing with Flexibility, Dynamic Strategies and Idioms in Mind.Computational Linguistics 15(1): 1–18.

    Google Scholar 

  • Stock, O. (1991). Natural Language and Exploration of an Information Space: the ALFRESCO Interactive System. In Proceedings ofThe IJCAI-91, 972–978. Sydney, Australia: Morgan Kaufmann.

    Google Scholar 

  • Strapparava, C. (1991). From Scopings to Interpretation: The Semantic Interpretation within the ALFRESCO System. In Ardizzone, Gaglio & Sorbello (eds.)Trends in Artificial Intelligence, Proceedings of The Second Congress of the Italian Association for Artificial Intelligence, 281–290. Springer-Verlag.

  • Stringa, L. (1990). An Integrated Approach to Artificial Intelligence.IRST-Technical Report 9012-11. IRST: Trento, Italy.

  • Suchman, L. A. (1987).Plans and Situated Action. Cambridge University Press.

  • Wahlster, W. (1988). User and Discourse Models for Multimodal Communication. In Sullivan, J. W. & Tyler, S. W. (eds.)Architectures for Intelligent Interfaces: Elements and Prototypes. Addison-Wesley.

  • Wahlster, W., André, E., Bandyopadhyay, S., Graf, W. & Rist, T. (1992). WIP: The Coordinated Generation of Multimodal Presentations from a Common Representation. In Slack, J., Ortony, A. & Stock, O. (eds.)Communication from Artificial Intelligence Perspective; Theoretical and Applied Issues, 121–143. Springer Verlag.

  • Weiser, M. (1991). The Computer for the 21st Century.Scientific American 265(3): 66.

    Google Scholar 

  • Zancanaro, M., Stock, O. & Strapparava, C. (1993). Dialogue Cohesion Sharing and Adjusting in an Enhanced Multimodal Environment. In Proceedings ofThe IJCAI-93, 1230–1236. Chambery, France: Morgan Kaufmann.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Additional information

This is a revised version of an invited talk that was delivered at the10th European Conference on Artificial Intelligence in Vienna, and published in the Proceedings (B. Neuman, ed.), John Wiley & Sons, 1992, pp. 853–862.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Stock, O. A third modality of natural language?. Artif Intell Rev 9, 129–146 (1995). https://doi.org/10.1007/BF00849176

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00849176

Key words

Navigation