Kenrick Kin
Björn Hartmann
ABSTRACT
Current multitouch frameworks require application developers to write recognition code for custom gestures; this code is split across multiple event-handling callbacks. As the number of custom gestures grows it becomes increasingly difficult to 1) know if new gestures will conflict with existing gestures, and 2) know how to extend existing code to reliably recognize the complete gesture set. Proton is a novel framework that addresses both of these problems. Using Proton, the application developer declaratively specifies each gesture as a regular expression over a stream of touch events. Proton statically analyzes the set of gestures to report conflicts, and it automatically creates gesture recognizers for the entire set. To simplify the creation of complex multitouch gestures, Proton introduces gesture tablature, a graphical notation that concisely describes the sequencing of multiple interleaved touch actions over time. Proton contributes a graphical editor for authoring tablatures and automatically compiles tablatures into regular expressions. We present the architecture and implementation of Proton, along with three proof-of-concept applications. These applications demonstrate the expressiveness of the framework and show how Proton simplifies gesture definition and conflict resolution.
Supplemental Material
- Adya, A., Howell, J., Theimer, M., Bolosky, W. J., and Douceur, J. R. Cooperative task management without manual stack management. Proc. USENIX 2002 (2002), 289--302. Google Scholar
Digital Library
- Appert, C., and Beaudouin-Lafon, M. SwingStates: adding state machines to the swing toolkit. Proc. UIST 2006 (2006), 319--322. Google ScholarDigital Library
- Apple. iOS. http://developer.apple.com/technologies/ios.Google Scholar
- Autodesk. SketchBook Pro. http://usa.autodesk.com/ adsk/servlet/pc/item?siteID=123112&id=15119465.Google Scholar
- Blackwell, A. SWYN: A Visual Representation for Regular Expressions. Morgan Kauffman, 2000, 245--270.Google Scholar
- Bleser, T., and Foley, J. D. Towards specifying and evaluating the human factors of user-computer interfaces. Proc. CHI 1982 (1982), 309--314. Google ScholarDigital Library
- Brzozowski, J. A. Derivatives of regular expressions. Journal of the ACM 11 (1964), 481--494. Google ScholarDigital Library
- De Nardi, A. Grafiti: Gesture recognition management framework for interactive tapletop interfaces. Master's thesis, University of Pisa, Italy, 2008.Google Scholar
- Echtler, F., and Klinker, G. A multitouch software architecture. Proc. NordiCHI 2008 (2008), 463--466. Google ScholarDigital Library
- Fraunhofer-Institute for Industrial Engineering. MT4j - Multitouch for Java. http://www.mt4j.org.Google Scholar
- Gibbon, D., Gut, U., Hell, B., Looks, K., Thies, A., and Trippel, T. A computational model of arm gestures in conversation. Proc. Eurospeech 2003 (2003), 813--816.Google Scholar
- Google. Android. http://www.android.com.Google Scholar
- Hansen, T. E., Hourcade, J. P., Virbel, M., Patali, S., and Serra, T. PyMT: a post-WIMP multi-touch user interface toolkit. Proc. ITS 2009 (2009), 17--24. Google ScholarDigital Library
- Henry, T. R., Hudson, S. E., and Newell, G. L. Integrating gesture and snapping into a user interface toolkit. Proc. UIST 1990 (1990), 112--122. Google ScholarDigital Library
- Hudson, S. E., Mankoff, J., and Smith, I. Extensible input handling in the subArctic toolkit. Proc. CHI 2005 (2005), 381--390. Google ScholarDigital Library
- Jacob, R. J. K. Executable specifications for a human-computer interface. Proc. CHI 1983 (1983), 28--34. Google ScholarDigital Library
- Jacob, R. J. K. A specification language for direct-manipulation user interfaces. TOG 5, 4 (October 1986), 283--317. Google ScholarDigital Library
- Jacob, R. J. K., Deligiannidis, L., and Morrison, S. A software model and specification language for non-WIMP user interfaces. TOCHI 6, 1 (1999), 1--46. Google ScholarDigital Library
- Kammer, D., Freitag, G., Keck, M., and Wacker, M. Taxonomy and overview of multi-touch frameworks: Architecture, scope and features. Workshop on Engineering Patterns for Multitouch Interfaces (2010).Google Scholar
- Kammer, D., Wojdziak, J., Keck, M., and Taranko, S. Towards a formalization of multi-touch gestures. Proc. ITS 2010 (2010), 49--58. Google ScholarDigital Library
- Khandkar, S. H., and Maurer, F. A domain specific language to define gestures for multi-touch applications. 10th Workshop on Domain-Specific Modeling (2010). Google ScholarDigital Library
- Kin, K., Miller, T., Bollensdorff, B., DeRose, T., Hartmann, B., and Agrawala, M. Eden: A professional multitouch tool for constructing virtual organic environments. Proc. CHI 2011 (2011), 1343--1352. Google ScholarDigital Library
- Lao, S., Heng, X., Zhang, G., Ling, Y., and Wang, P. A gestural interaction design model for multi-touch displays. Proc. British Computer Society Conference on Human-Computer Interaction (2009), 440--446. Google ScholarDigital Library
- Lu, H., and Li, Y. Gesture Coder: A tool for programming multi-touch gestures by demonstration. Proc. CHI 2012 (2012). Google ScholarDigital Library
- Mankoff, J., Hudson, S. E., and Abowd, G. D. Interaction techniques for ambiguity resolution in recognition-based interfaces. Proc. UIST 2000 (2000), 11--20. Google ScholarDigital Library
- Mankoff, J., Hudson, S. E., and Abowd, G. D. Providing integrated toolkit-level support for ambiguity in recognition-based interfaces. Proc. CHI 2000 (2000), 368--375. Google ScholarDigital Library
- MERL. DiamondTouch. http://merl.com/projects/DiamondTouch.Google Scholar
- Microsoft. Windows 7. http://www.microsoft.com/en-US/ windows7/products/home.Google Scholar
- Myers, B. A. A new model for handling input. ACM Trans. Inf. Syst. 8, 3 (1990), 289--320. Google ScholarDigital Library
- Newman, W. M. A system for interactive graphical programming. Proc. AFIPS 1968 (Spring) (1968), 47--54. Google ScholarDigital Library
- NUI Group. Touchlib. http://nuigroup.com/touchlib.Google Scholar
- Olsen, D. Building Interactive Systems: Principles for Human-Computer Interaction. Course Technology Press, Boston, MA, United States, 2009, 43--66. Google ScholarDigital Library
- Olsen, Jr., D. R., and Dempsey, E. P. Syngraph: A graphical user interface generator. Proc. SIGGRAPH 1983 (1983), 43--50. Google ScholarDigital Library
- Raskin, J. The Humane Interface. Addison Wesley, 2000.Google Scholar
- Scholliers, C., Hoste, L., Signer, B., and De Meuter, W. Midas: a declarative multi-touch interaction framework. Proc. TEI 2011 (2011), 49--56. Google ScholarDigital Library
- Schwarz, J., Hudson, S. E., Mankoff, J., and Wilson, A. D. A framework for robust and flexible handling of inputs with uncertainty. Proc. UIST 2010 (2010), 47--56. Google ScholarDigital Library
- Shaer, O., and Jacob, R. J. K. A specification paradigm for the design and implementation of tangible user interfaces. TOCHI 16, 4 (2009). Google ScholarDigital Library
- Signer, B., Kurmann, U., and Norrie, M. iGesture: A general gesture recognition framework. Proc. ICDAR 2007 (2007), 954--958. Google ScholarDigital Library
- Sipser, M. Introduction to the Theory of Computation, 1st ed. International Thomson Publishing, 1996, 46,70--76. Google ScholarDigital Library
- Siwgart, S. Easily write custom gesture recognizers for your tablet PC applications, November 2005. Microsoft Technical Report.Google Scholar
- Sparsh UI. http://code.google.com/p/sparsh-ui.Google Scholar
- Thompson, K. Regular expression search algorithm. Communications of the ACM 11, 6 (1968), 419--422. Google ScholarDigital Library
- Wobbrock, J. O., Myers, B. A., and Kembel, J. A. Edgewrite: A stylus-based text entry method designed for high accuracy and stability of motion. Proc. UIST 2003 (2003), 61--70. Google ScholarDigital Library
- Worth, C. D. xstroke. http://pandora.east.isi.edu/xstroke/usenix_2003.Google Scholar
Index Terms
- Proton: multitouch gestures as regular expressions
Recommendations
Proton++: a customizable declarative multitouch framework
UIST '12: Proceedings of the 25th annual ACM symposium on User interface software and technologyProton++ is a declarative multitouch framework that allows developers to describe multitouch gestures as regular expressions of touch event symbols. It builds on the Proton framework by allowing developers to incorporate custom touch attributes directly ...
Transactional Read-Modify-Write Without Aborts
Language-level transactions are said to provide “atomicity,” implying that the order of operations within a transaction should be invisible to concurrent transactions and thus that independent operations within a transaction should be safe to execute in ...
TouchTools: leveraging familiarity and skill with physical tools to augment touch interaction
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsThe average person can skillfully manipulate a plethora of tools, from hammers to tweezers. However, despite this remarkable dexterity, gestures on today's touch devices are simplistic, relying primarily on the chording of fingers: one-finger pan, two-...
Comments