Levent Burak Kara
ABSTRACT
A long standing challenge in pen-based computer interaction is the ability to make sense of informal sketches. A main difficulty lies in reliably extracting and recognizing the intended set of visual objects from a continuous stream of pen strokes. Existing pen-based systems either avoid these issues altogether, thus resulting in the equivalent of a drawing program, or rely on algorithms that place unnatural constraints on the way the user draws. As one step toward alleviating these difficulties, we present an integrated sketch parsing and recognition approach designed to enable natural, fluid, sketch-based computer interaction. The techniques presented in this paper are oriented toward the domain of network diagrams. In the first step of our approach, the stream of pen strokes is examined to identify the arrows in the sketch. The identified arrows then anchor a spatial analysis which groups the uninterpreted strokes into distinct clusters, each representing a single object. Finally, a trainable shape recognizer, which is informed by the spatial analysis, is used to find the best interpretations of the clusters. Based on these concepts, we have built SimuSketch, a sketch-based interface for Matlab's Simulink software package. An evaluation of SimuSketch has indicated that even novice users can effectively utilize our system to solve real engineering problems without having to know much about the underlying recognition techniques.
- Fevzi Alimoglu and Ethem Alpaydin. Combining multiple representations for pen-based handwritten digit recognition. ELEKTRIK: Turkish Journal of Electrical Engineering and Computer Sciences, 9(1): 1--12, 2001.Google Scholar
- Christine Alvarado. A Natural Sketching Environment: Bringing the Computer into Early Stages of Mechanical Design. Master thesis, MIT, 2000.Google Scholar
- Christine Alvarado. Dynamically constructed bayesian networks for sketch understanding. Technical report, MIT Project Oxygen Student Workshop Abstracts, 2003.Google Scholar
- Christine Alvarado and Randall Davis. Resolving ambiguities to create a natural sketch based interface. In IJCAI-2001, 2001. Google ScholarDigital Library
- Ajay Apte, Van Vo, and Takayuki Dan Kimura. Recognizing multistroke geometric shapes: An experimental evaluation. In UIST93, pages 121--128, 1993. Google ScholarDigital Library
- Chris Calhoun, Thomas F Stahovich, Tolga Kurtoglu, and Levent Burak Kara. Recognizing multi-stroke symbols. In AAAI Spring Symposium on Sketch Understanding, pages 15--23, 2002.Google Scholar
- Gennaro Costagliola and Vincenzo Deufemia. Visual language editors based on lr parsing techniques. In 8th International Workshop on Parsing Technologies (IWPT'03), Nancy, France, 2003.Google Scholar
- Marie-Pierre Dubuisson and Anil K Jain. A modified hausdorff distance for object matching. In 12th International Conference on Pattern Recognition, pages 566--568, Jerusalem, Israel, 1994.Google ScholarCross Ref
- Lee D Erman, Frederick Hayes-Roth, Victor R Lesser, and D Raj Reddy. The hearsay-ii speech understanding system: Integrating knowldge to resolve uncertainty. Computing Surveys, 12(2):213--253, 1980. Google ScholarDigital Library
- Michael Fligner, Joseph Verducci, Jeff Bjoraker, and Paul Blower. A new association coefficient for molecular dissimilarity. In The Second Joint Sheffield Conference on Chemoinformatics, Sheffield, England, 2001.Google Scholar
- Manueal J Fonseca, Cesar Pimentel, and Jaoquim A Jorge. Cali-an online scribble recognizer for calligraphic interfaces. In AAAI Spring Symposium on Sketch Understanding, pages 51--58, 2002.Google Scholar
- Manuel J Fonseca and Joaquim A Jorge. Using fuzzy logic to recognize geometric shapes interactively. In Proceedings of the 9th Int. Conference on Fuzzy Systems (FUZZ-IEEE 2000). San Antonio, USA, 2000.Google ScholarCross Ref
- W Eric L Grimson. The combinatorics of heuristic search termination for object recognition in cluttered environments. IEEE PAMI, 13(9):920--935, 1991. Google ScholarDigital Library
- Jason I Hong and James A Landay. Satin: A toolkit for informal ink-based applications. In ACM UIST 2000 User Interfaces and Software Technology, pages 63--72, San Diego, CA, 2000. Google ScholarDigital Library
- Heloise Hse and A. Richard Newton. Sketched symbol recognition using zernike moments. Technical report, EECS, University of California, 2003.Google Scholar
- David W Jacobs. The use of grouping in visual object recognition. Technical Report Technical Report 1023, MIT AI Lab, 1988. Google Scholar
- T D Kimura, A Apte, and S Sengupta. A graphic diagram editor for pen computers. Software Concepts and Tools, pages 82--95, 1994.Google Scholar
- Tolga Kurtoglu and Thomas F Stahovich. Interpreting schematic sketches using physical reasoning. In AAAI Spring Symposium on Sketch Understanding, pages 78--85, 2002.Google Scholar
- Ernst Kussul and Tatyana Baidyk. Improved method of handwritten digit recognition tested on mnist database. In 15th International Conference on Vision Interface, Calgary, Canada, 2002.Google Scholar
- James A Landay and Brad A Myers. Sketching interfaces: Toward more human interface design. IEEE Computer, 34(3):56--64, 2001. Google ScholarDigital Library
- Y LeCun, L D Jackel, L Bottou, A Brunot, C Cortes, J S Denker, H Drucker, I Guyon, U A Muller, E Sackinger, P Simard, and V Vapnik. Comparison of learning algorithms for handwritten digit recognition. In International Conference on Artificial Neural Networks, pages 53--60, Paris, 1995.Google Scholar
- James Lin, Mark W. Newman, Jason I. Hong, and James A. Landay. Denim: Finding a tighter fit between tools and practice for web site design. In CHI Letters: Human Factors in Computing Systems, pages 510--517. ACM Press, 2000. Google ScholarDigital Library
- Jennifer Mankoff, Gregory D. Abowd, and Scott E Hudson. Oops: a toolkit supporting mediation techniques for resolving ambiguity in recognition-based interfaces.Computers and Graphics, 24(6):819--834, 2000.Google ScholarCross Ref
- Nicholas E Matsakis. Recognition of Handwritten Mathematical Expressions. Master thesis, MIT, 1999.Google Scholar
- Shankar Narayanaswamy. Pen and Speech Recognition in the User Interface for Mobile Multimedia Terminals. Ph.d. thesis, University of California at Berkeley, 1996. Google ScholarDigital Library
- Omer Faruk Ozer, Oguz Ozun, C Oncel Tuzel, Volkan Atalay, and A Enis Cetin. Vision-based single-stroke character recognition for wearable computing. IEEE Intelligent Systems and Applications, 16(3):33--37, 2001. Google ScholarDigital Library
- Dean Rubine. Specifying gestures by example. Computer Graphics, 25:329--337, 1991. Google ScholarDigital Library
- W J Rucklidge. Efficient Visual Recognition Using the Hausdorff Distance. Number 1173 Lecture Notes in computer Science,. Springer-Verlag, Berlin, 1996. Google ScholarDigital Library
- Eric Saund, James Mahoney, David Fleet, Dan Larner, and Edward Lank. Perceptual organisation as a foundation for intelligent sketch editing. In AAAI Spring Symposium on Sketch Understanding, pages 118--125, 2002.Google Scholar
- Tevfik Metin Sezgin. Generic and HMM based approaches to freehand sketch recognition. Technical report, MIT Project Oxygen Student Workshop Abstracts, 2003.Google Scholar
- Michael Shilman, Hanna Pasula, Stuart Russell, and Richard Newton. Statistical visual language models for ink parsing. In AAAI Spring Symposium on Sketch Understanding, pages 126--132, 2002.Google Scholar
- Jack D Tubbs. A note on binary template matching. Pattern Recognition, 22(4):359--365, 1989. Google ScholarDigital Library
- H Yasuda, K Takahashi, and T Matsumoto. A discrete HMM for online handwriting recognition. International Journal of Pattern Recognition and Articial Intelligence, 14(5):675--688, 2000.Google ScholarCross Ref
Index Terms
- Hierarchical parsing and recognition of hand-sketched diagrams
Recommendations
Hierarchical parsing and recognition of hand-sketched diagrams
UIST '04: Proceedings of the 17th annual ACM symposium on User interface software and technologyA long standing challenge in pen-based computer interaction is the ability to make sense of informal sketches. A main difficulty lies in reliably extracting and recognizing the intended set of visual objects from a continuous stream of pen strokes. ...
An efficient graph-based recognizer for hand-drawn symbols
We describe a trainable, multi-stroke symbol recognizer for pen-based user interfaces. The approach is insensitive to orientation, non-uniform scaling, and drawing order. Symbols are represented internally as attributed relational graphs describing both ...
Combining geometry and domain knowledge to interpret hand-drawn diagrams
One main challenge in building interpreters for hand-drawn sketches is the task of parsing a sketch to locate the individual symbols. Many existing pen-based systems avoid this problem by requiring the user to explicitly indicate the partitioning of the ...
Comments