Skip to main content
SpringerLink
Log in

A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets

  • Original Paper
  • Published:
International Journal on Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

We present a new approach for parsing two-dimensional input using relational grammars and fuzzy sets. A fast, incremental parsing algorithm is developed, motivated by the two-dimensional structure of written mathematics. The approach reports all identifiable parses of the input. The parses are represented as a fuzzy set, in which the membership grade of a parse measures the similarity between it and the handwritten input. To identify and report parses efficiently, we adapt and apply existing techniques such as rectangular partitions and shared parse forests, and introduce new ideas such as relational classes and interchangeability. We also present a correction mechanism that allows users to navigate parse results and choose the correct interpretation in case of recognition errors or ambiguity. Such corrections are incorporated into subsequent incremental recognition results. Finally, we include two empirical evaluations of our recognizer. One uses a novel user-oriented correction count metric, while the other replicates the CROHME 2011 math recognition contest. Both evaluations demonstrate the effectiveness of our proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Álvaro, F., Sánchez, J.-A., Benedí, J.-M.: Recognition of printed mathematical expressions using two-dimensional stochastic context-free grammars. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 1225–1229 (2011)

  2. Anderson, R.H.: Syntax-directed Recognition of Hand-printed Two-dimensional Mathematics, Ph.D. thesis, Harvard University (1968)

  3. Awal, A.-M., Mouchère, H., Viard-Gaudin, C.: The problem of handwritten mathematical expression recognition evaluation. In: Proceedings of the International Conference on Frontiers in Handwriting Recognition, pp. 646–651 (2010)

  4. Belaid A., Haton J.-P.: A syntactic approach for handwritten mathematical formula recognition. IEEE Trans. Pattern Anal. Mach. Intell. 6(1), 105–111 (1984)

    Article  Google Scholar 

  5. Blostein D., Grbavec A.: Recognition of mathematical notation. In: Wang, P.S.P., Bunke, H. (eds) Handbook on Optical Character Recognition in Document Analysis, pp. 557–582. World Scientific, Singapore (1996)

    Google Scholar 

  6. Blostein D.: Math-literate computers, Calculemus/MKM. In: Carette, J., Dixon, L., Coen, C.S., Watt, S.M. (eds) Lecture Notes in Computer Science, vol. 5625, pp. 2–13. Springer, Berlin (2009)

    Google Scholar 

  7. Caraballo S.A., Charniak E.: New figures of merit for best-first probabilistic chart parsing. Comput. Linguist. 24(2), 275–298 (1998)

    Google Scholar 

  8. Chan, K.-F., Yeung, D.-Y.: Error detection, error correction and performance evaluation in on-line mathematical expression recognition. In: On-Line Mathematical Expression Recognition, Pattern Recognition (1999)

  9. Chan K.-F., Yeung D.-Y.: Mathematical expression recognition: a survey. Int. J. Doc. Anal. Recogn. 3, 3–15 (1999)

    Article  Google Scholar 

  10. Chou, P.A.: Recognition of equations using a two-dimensional stochastic context-free grammar. In: Proceedings of the SPIE, Visual Communication and Image Processing IV, vol. 1199, pp. 852–863 (1989)

  11. Costagliola G., De Lucia A., Orefice S., Tortora G.: Positional grammars: a formalism for lr-like parsing of visual languages. In: Marriott, K., Meyer, B. (eds) Visual Language Theory, pp. 171–192. Springer, New York (1998)

    Chapter  Google Scholar 

  12. Fitzgerald, J.A., Geiselbrechtinger, F., Kechadi, T.: Mathpad: A fuzzy logic-based recognition system for handwritten mathematics, Document Analysis and Recognition. ICDAR 2007. Ninth International Conference on, vol. 2, Sept. 2007, pp. 694–698 (2007)

  13. Garain U., Chaudhuri B.B.: Recognition of online handwritten mathematical expressions. Syst. Man. Cybern. Part B: Cybern. IEEE Trans. 34(6), 2366–2376 (2004)

    Google Scholar 

  14. Garain U., Chaudhuri B.: A corpus for ocr research on mathematical expressions. Int. J. Doc. Anal. Recognit. 7(4), 241–259 (2005)

    Article  Google Scholar 

  15. Grune D., Jacobs C.J.H.: Parsing Techniques: A Practical Guide, 2 edn. Springer, Berlin (2008)

    Google Scholar 

  16. Hull, J.: Recognition of mathematics using a two-dimensional trainable context-free grammar, Master’s thesis, Massachusetts Institute of Technology (1996)

  17. Labahn, G., Lank, E., MacLean, S., Marzouk, M., Tausky, D.: Mathbrush: a system for doing math on pen-based devices. In: The Eighth IAPR Workshop on Document Analysis Systems (DAS), pp. 599–606 (2008)

  18. Bernard L.: Towards a Uniform Formal Framework for Parsing, Current Issues in Parsing Technology, pp. 153–171. Kluwer, Dordrecht (1991)

    Google Scholar 

  19. Laviola, Jr. J.J.: Mathematical Sketching: A New Approach to Creating and Exploring Dynamic Illustrations, Ph.D. thesis, Brown University (2005)

  20. Lee E.T., Zadeh L.A.: Note on fuzzy languages. In: Klir, G.J., Yuan, B. (eds) Fuzzy Sets, Fuzzy Logic, and Fuzzy Systems, pp. 69–82. World Scientific, Singapore (1996)

    Google Scholar 

  21. Li, C., Miller, T.S., Zeleznik, R.C., LaViola Jr. J.J.: Algosketch: Algorithm sketching and interactive computation. In: Proceedings of Sketch-Based Interfaces and Modeling (2008)

  22. Liang, P., Narasimhan, M., Shilman, M., Viola, P.: Efficient geometric algorithms for parsing in two dimensions, ICDAR ’05: Proceedings of the Eighth International Conference on Document Analysis and Recognition (Washington, DC, USA), IEEE Computer Society, pp. 1172–1177 (2005)

  23. Liu, W., Zhang, L., Tang, L., Dori, D.: Cost evaluation of interactively correcting recognized engineering drawings, Lecture Notes in Computer Science, no. 1941, pp. 329–334 (2000)

  24. MacLean S., Labahn G., Lank E., Marzouk M., Tausky D.: Grammar-based techniques for creating ground-truthed sketch corpora. Int. J. Doc. Anal. Recogn. 14, 65–74 (2011)

    Article  Google Scholar 

  25. MacLean, S.: Tools for the efficient generation of hand-drawn corpora based on context-free grammars. In: Third International Workshop on Pen-Based Mathematics Computing, http://www.orcca.on.ca/conferences/cicm09/workshops/PenMath/programme-hand.html (2009)

  26. MacLean, S., Labahn, G.: Elastic matching in linear time and constant space. In: Proceedings of Ninth IAPR International Workshop on Document Analysis Systems, (Short paper), pp. 551–554 (2010)

  27. MacLean, S., Labahn, G.: Recognizing handwritten mathematics via fuzzy parsing, Technical Report CS-2010-13, School of Computer Science, University of Waterloo (2010)

  28. Manning C.D., Schuetze H.: Foundations of Statistical Natural Language Processing. The MIT Press, Cambridge (1999)

    MATH  Google Scholar 

  29. Marriott K., Meyer B., Wittenburg K.B.: A survey of visual language specification and recognition. In: Marriott, K., Meyer, B. (eds) Visual Language Theory, pp. 5–85. Springer, New York (1998)

    Chapter  Google Scholar 

  30. Miller, E.G., Viola, P.A.: Ambiguity and constraint in mathematical expression recognition. In: Proceedings of the Fifteenth National/tenth Conference on Artificial Intelligence/Innovative Applications of Artificial Intelligence, pp. 784–791 (1998)

  31. Mouchère, H., Viard-Gaudin, C., Garain, U., Kim, D.H., Kim, J.H.: Crohme2011: Competition on recognition of online handwritten mathematical expressions. In: Proceedings of the 11th International Conference on Document Analysis and Recognition (2011)

  32. Rhee T.H., Kim J.H.: Efficient search strategy in structural analysis for handwritten mathematical expression recognition. Pattern Recogn. 42(12), 3192–3201 (2009)

    Article  MATH  Google Scholar 

  33. Rutherford, I.: Structural analysis for pen-based math input systems, Master’s thesis, David R. Cheriton School of Computer Science, University of Waterloo (2005)

  34. Sain K., Dasgupta A., Garain U.: Emers: a tree matching-based performance evaluation of mathematical expression recognition systems. IJDAR 14(1), 75–85 (2011)

    Article  Google Scholar 

  35. Shi, Y., Li, H., Soong, F.K.: A unified framework for symbol segmentation and recognition of handwritten mathematical expressions. In: Ninth International Conference on Document Analysis and Recognition, pp. 854–858 (2007)

  36. Tomita M.: Parsing 2-dimensional languages. In: Tomita, M. (eds) Current Issues in Parsing Technology, pp. 277–289. Kluwer, Dordrecht (1991)

    Chapter  Google Scholar 

  37. Toyota, S., Uchida, S., Suzuki, M.: Structural analysis of mathematical formulae with verification based on formula description grammar. Doc. Anal. Syst. VII:153–163 (2006)

    Google Scholar 

  38. Unger S.H.: A global parser for context-free phrase structure grammars. Commun. ACM 11, 240–247 (1968)

    Article  MathSciNet  Google Scholar 

  39. Winkler, H.-J., Fahrner, H., Lang, M.: A soft-decision approach for structural analysis of handwritten mathematical expressions. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing, pp. 2459–2462 (1995)

  40. Zadeh L.A.: Fuzzy sets. Inf. Control 8, 338–353 (1965)

    Article  MathSciNet  MATH  Google Scholar 

  41. Zanibbi R., Blostein D., Cordy J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1455–1467 (2002)

    Article  Google Scholar 

  42. Zhang, L., Blostein, D., Zanibbi, R.: Using fuzzy logic to analyze superscript and subscript relations in handwritten mathematical expressions. In: Proceedings of Eighth International Conference on Document Analysis and Recognition, vol. 2, pp. 972–976 (2005)

Download references

Author information

Authors and Affiliations

  1. Cheriton School of Computer Science, University of Waterloo, 200 University Ave. W., Waterloo, ON, N2L 3G1, Canada

    Scott MacLean & George Labahn

Authors
  1. Scott MacLean
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. George Labahn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Scott MacLean.

Rights and permissions

Reprints and permissions

About this article

Cite this article

MacLean, S., Labahn, G. A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. IJDAR 16, 139–163 (2013). https://doi.org/10.1007/s10032-012-0184-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-012-0184-x

Keywords

Search

Navigation