P. Bottoni
M. F. Costabile
Abstract
Computers are increasingly being seen not only as computing tools but more so as communication tools, thus placing special emphasis on human-computer interaction (HCI). In this article, the focus is on visual HCI, where the messages exchanged between human and computer are images appearing on the computer screen, as usual in current popular user interfaces. We formalize interactive sessions of a human-computer dialogue as a structured set of legal visual sentences, i.e., as a visual language, and show how rewriting systems can be generalized to specify both the pictorial and the computational aspects of visual languages. To this end, Visual Conditional Attributed Rewriting (VCARW) systems are introduced, and use for specification of visual languages. These specifications are given as inputs to a procedure illustrated in the article as a system of algorithms, which automatically generates control mechanisms of the interaction, thus favoring the design of more reliable and usable systems.
- ABOWD, G. AND BEALE, R. 1991. Users, systems and interfaces: A unifying framework for interaction. In Proceedings of HCI '91: People and Computers VI, D. Diaper and N. Hammond, Eds. 73-87.Google Scholar
- ANDRIES, M., ENGELS, G., AND REKERS, g. 1998. How to represent a visual specification. In Visual Language Theory, K. Marriott and B. Meyer, Eds. Springer, 245-259. Google ScholarDigital Library
- AREFI, F., MILANI, M., AND STARY, C. 1991. Towards customized user interface design environments. In Proceedings of the IEEE Workshop on Visual Languages. IEEE, New York, 146-151.Google ScholarCross Ref
- BURNETT, M. AND AMBLER, A. 1994. Interactive visual data abstraction in a declarative visual programming language. J. Vis. Lang. Comput. 5, 26-60.Google ScholarCross Ref
- BRANCHEAU, J. C. AND BROWN, C.V. 1993. The management of end-user computing: Status and direction. ACM Comput. Surv. 25, 4, 437-482. Google ScholarDigital Library
- BIANCHI, N., BOTTONI, P., MussIo, P., AND PROTTI, M. 1993. Cooperative visual environments for the design of effective visual systems. J. Vis. Lang. Comput. 4, 357-382.Google ScholarCross Ref
- BIANCHI, N., BOTTONI, P., SPINU, C., GARBAY, C., AND MUSSIO, P. 1998. Situated image understanding in a multi-agent framework. Int. J. Pattern Recog. Art. Intell. 12, 5, 595-624.Google ScholarCross Ref
- BOTTONI, P., CHANG, S.-K., COSTABILE, M. F., LEVIALDI, S., AND MUSSIO, P. 1998a. On the specification of dynamic visual language. In Proceedings of the IEEE Symposium on Visual Languages. IEEE, New York, 14-21. Google ScholarDigital Library
- BOTTONI, P., COSTABILE, M.F., LEVIALDI, S., AND MUSSIO, P. 1996a. Visual conditional attributed rewriting systems in visual language specification. In Proceedings of the IEEE Symposium on Visual Languages. IEEE, New York, 156-163. Google ScholarDigital Library
- BOTTONI, P., COSTABILE, M. F., LEVIALDI, S., AND MUSSIO, P. 1996b. A visual approach to HCI. ACM SIGCHI Bull. 28, 3, 50-55. Google ScholarDigital Library
- BOTTONI, P., COSTABILE, M. F., LEVIALDI, S., AND MUSSIO, P. 1997. Defining visual languages for interactive computing. IEEE Trans. Syst. Man. Cybernet. A-27, 6, 773-782. Google ScholarDigital Library
- BOTTONI, P., COSTABILE, M. F., LEVIALDI, S., AND MUSSIO, P. 1998b. Specification of visual languages as means for interaction. In Visual Language Theory, K. Marriott and B. Meyer, Eds. Springer, 353-375. Google ScholarDigital Library
- BOTTONI, P., COSTABILE, M. F., LEVIALDI, S., AND MUSSIO, P. 1998c. Specifying dialog control in visual interactive systems. J. Vis. Lang. Comput. 9, 5, 553-564.Google ScholarCross Ref
- BOTTONI, P., MussIo, P., OLIVIERI, B., AND PROTTI, M. 1998d. A completely visual environment for agent-based computing. In Proceedings of AVI '98, T. Catarci, M. F. Costabile, G. Santucci, and L. Tarantino, Eds. ACM Press, New York, 261-263. Google ScholarDigital Library
- BOYLAND, J.T. 1996. Conditional attribute grammars. ACM Trans. Program. Lang. Syst. 18, 1, 73-108. Google ScholarDigital Library
- CARLE, A. AND POLLOCK, L. 1996. On the optimality of change propagation for incremental evaluation of hierarchical attribute grammars. ACM Trans. Program. Lang. Syst. 18, 1, 16-29. Google ScholarDigital Library
- CHANG, S. K. AND MUSSIO, P. 1996. Customized visual language design. In Proceedings of the 8th International Conference on Software Engineering and Knowledge Engineering (SEKE '96). 553-562.Google Scholar
- CHANG, S. K., TAUBER, M. J., Yu, B., AND YU, J.S. 1988. The SIL-Icon compiler--An objectoriented system generator. Int. J. Patt. Recognit. Art. Intell. 2, 241-273.Google ScholarCross Ref
- CHANG, S. K., TORTORA, G., Yu, B., AND GUERICIO, A. 1987. Icon purity--Towards a formal definition of Icon. Int. J. Patt. Recognit. Art. Intell. 1, 377-392.Google ScholarCross Ref
- CHOK, S. S. AND MARRIOTT, Z. 1995. Automatic construction of user interfaces from constraint multiset grammars. In Proceedings of the IEEE Symposium on Visual Languages. IEEE, New York, 242-249. Google ScholarDigital Library
- COSTAGLIOLA, G., TORTORA, G., OREFICE, S., AND DELUCIA, A. 1995. Automatic generation of visual programming environments. IEEE Comput. 28, 3, 56-66. Google ScholarDigital Library
- COSTAGLIOLA, G., TORTORA, G., OREFICE, S., AND DELUCIA, A. 1997. A parsing methodology for the implementation of visual systems. IEEE Trans. Softw. Eng. 23, 11, 777-779. Google ScholarDigital Library
- COUTAZ, J. 1991. Architectural design for user interfaces. In Proceedings of the 3rd Conference of Software Engineering (ESEC '91). 7-22. Google ScholarDigital Library
- DASSOW, J. AND PAUN, G. 1989. Regulated Rewriting in Formal Language Theory. Springer. Google ScholarDigital Library
- DIx, A. 1991. Formal Methods for Interactive Systems. Academic Press.Google Scholar
- DUKE, K. J. AND HARRISON, M.D. 1995. Interaction and task requirements. In DSV-IS '95: Eurographics Workshop on Design, Specification and Verification of Interactive Systems, P. Palanque and R. Bastide, Eds. Springer, 54-75.Google ScholarCross Ref
- FACONTI, G. P. AND DUKE, D.J. 1996. Device models. In Design, Specification and Verification of Interactive Systems, F. Bodart and J. Vanderdonckt, Eds. Springer, 73-91.Google Scholar
- FOLEY, J. D., VAN DAM, A., FEINER, S. K., HUGHES, J. F., AND PHILLIPS, R.L. 1993. Introduction to Computer Graphics. Addison-Wesley. Google ScholarDigital Library
- GOLIN, E.J. 1991. Parsing visual languages with picture layout grammars. J. Vis. Lang. Comput. 2, 371-394.Google ScholarDigital Library
- GREEN, M. 1985. Report on dialogue specification tools. In User Interface Management Systems. Springer, 9-20.Google Scholar
- HARRISON, M.D. AND DIX, A. 1990. A state model for direct manipulation. In Formal Methods in HIC, M. D. Harrison and H. W. Thimbleby, Eds. Cambridge University Press, 129-151. Google ScholarDigital Library
- HELM, R. AND MARRIOTT, K. 1991. A declarative specification and semantics for visual languages. J. Vis. Lang. Comput. 2, 311-332.Google ScholarDigital Library
- HIRAKAWA, M., NISHIMURA, Y., KADO, M., AND ICHIKAWA, T. 1991. Interpretation of icon overlapping in iconic programming. In Proceedings of the IEEE Workshop on Visual Languages. IEEE, New York, 254-259.Google ScholarCross Ref
- KAHN, K. AND SARASWAT, V.A. 1990. Complete visualisations of concurrent programs and their executions. In Proceedings of the IEEE Workshop on Visual Languages. IEEE, New York, 7-15.Google Scholar
- KATAYAMA, T. 1984. Translation of attribute grammars into procedures. ACM Trans. Program. Lang. Syst. 6, 3, 345-369. Google ScholarDigital Library
- MARRIOTT, K., MEYER, B., AND WITTENBURG, K. 1998. A survey of visual language specification and recognition. In Visual Language Theory, K. Marriott and B. Meyer, Eds. Springer, 5-85. Google ScholarDigital Library
- MESEGUER, J. 1992. Conditional rewriting logic as a unified model of concurrency. Theoret. Comput. Sci. 96, 1, 73-155. Google ScholarDigital Library
- MEVEL, A. AND GUEGUEN, T. 1987. Smalltalk-80. MacMillan Education.Google Scholar
- MINAS, M. 1997. Diagram editing with hypergraph parser support. In Proceedings of the IEEE Symposium on Visual Languages. IEEE, New York, 226-233. Google ScholarDigital Library
- MussIo, P., FINADRI, M., GENTINI, P., AND COLOMBO, F. 1992. A bootstrap approach to visual user-interface design and development. Vis. Comput. 8, 2, 75-93. Google ScholarDigital Library
- MussIo, P., PIETROGRANDE, M., AND PROTTI, M. 1991. Simulation ofhepatological models: A study in visual interactive exploration of scientific problems. J. Vis. Lang. Comput. 2, 75-95.Google ScholarDigital Library
- NAJORK, M.A. AND KAPLAN, S.M. 1993. Specifying visual languages with conditional set rewrite systems. In Proceedings of the IEEE Workshop on Visual Languages. IEEE, New York, 12-18.Google ScholarCross Ref
- NORMAN, D.A. 1988. The Psychology of Everyday Things. Basic Books.Google Scholar
- PAPAKONSTANTINOU, a. 1979. A poor man's realization of attribute grammars. Softw. Pract. Exper. 9, 9, 719-728.Google ScholarCross Ref
- PETERSON, J.L. 1981. Petri Net Theory and the Modeling of Systems. Prentice-Hall. Google ScholarDigital Library
- REKERS, J. AND SCHUERR, A. 1995. A graph grammar approach to graphical parsing. In Proceedings of the IEEE Symposium on Visual Languages. IEEE, New York, 195-202. Google ScholarDigital Library
- REKERS, J. AND SCHUERR, A. 1996. A graph-based framework for the implementation of visual environments. In Proceedings of the IEEE Symposium on Visual Languages. IEEE, New York, 148-155. Google ScholarDigital Library
- REPENNING, A. AND SUMNER, T. 1995. Agentsheets: A medium for creating domain-oriented visual languages. IEEE Comput. 28, 3, 17-26. Google ScholarDigital Library
- SALOMAA, A. 1973. Formal Languages. Academic Press. Google ScholarDigital Library
- SMITH, D. C., CYPHER, A., AND SPORER, J. 1994. KidSim: Programming agents without a programming language. Commun. ACM 37, 7, 55-67. Google ScholarDigital Library
- SERRANO, J.A. 1995. The use of semantic constraints on diagram editors. In Proceedings of the IEEE Symposium on Visual Languages. IEEE, New York, 211-2167. Google ScholarDigital Library
- SHNEIDERMAN, B. 1992. Designing the User Interface. Addison Wesley. Google ScholarDigital Library
- TONDL, L. 1981. Problems of Semantics. Reidel.Google Scholar
- WANG, D., LEE, g. R., AND ZEEVAT, H. 1995. Reasoning with diagrammatic representations. In Diagrammatic Reasoning, J. Glasgow, N. H. Narayanan, and B. Chandrasekaran, Eds. AAAI Press, Menlo Park, CA, 339-393.Google Scholar
- WEGNER, P. 1995. Interaction as a basis for empirical computer science. ACM Comput. Surv. 27, 45-48. Google ScholarDigital Library
- WEGNER, P. 1997. Why interaction is more powerful than algorithms? Commun. ACM 40, 5, 80-91. Google ScholarDigital Library
- WITTENBURG, K. AND WEITZMAN, L. 1998. Relational grammars: Theory and practice in a visual language interface for process modeling. In Visual Language Theory, K. Marriott and B. Meyer, Eds. Springer, 193-217. Google ScholarDigital Library
Index Terms
- Specification and dialogue control of visual interaction through visual rewriting systems
Recommendations
From visual language specification to legal visual interaction
VL '97: Proceedings of the 1997 IEEE Symposium on Visual Languages (VL '97)A crucial requirement of visual interaction is to prevent users from performing incorrect operations. This requires the computer environment to trap illegal users' actions. On the other hand, users should have enough freedom in their interaction, ...
Visual Conditional Attributed Rewriting Systems in Visual Language Specification
VL '96: Proceedings of the 1996 IEEE Symposium on Visual LanguagesIn order to manage visual interaction problems, we deepen a formalisation of visual languages, based on the notion of visual sentence, introduced in a previous paper. Visual Conditional attributed rewriting systems (vCARWs) are proposed for the ...
Formalising visual languages
VL '95: Proceedings of the 11th International IEEE Symposium on Visual LanguagesWe propose a formalisation of visual languages which allows a uniform approach to satisfying the needs of pattern recognition, image generation, and visual reasoning faced in visual human-computer communication. Such needs comprise managing the full ...
Reviews
Claudiu Popescu
An important aspect of computer usage is the good interaction between the user and the computer. This paper presents the results of research into a formalization of visual interaction. The authors define a visual language and introduce visual conditional attribute rewriting systems. Most of the paper consists of the authors' mathematical formalization of the specification of the visual interaction. They include numerous definitions and theorems. The widely known Petri nets are used as an example of this formalization. An application of the formalization to specifying automata for dialogue control is presented and illustrated by pictures of the accompanying software. An appendix contains the formal definitions of the visual elements (alphabet, sentence, and language). The paper ends with numerous references. The authors take a deep theoretical approach to the field and cover high-level research. Effective graphics make this interesting paper, on a difficult subject, easier to read. The paper is appropriate for researchers and students in formal languages and automata. Additional material on applications of this theory would have been useful.
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments