Skip to main content
SpringerLink
Log in

Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer

  • Conference paper
  • First Online:
Document Analysis and Recognition – ICDAR 2021 (ICDAR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12822))

Included in the following conference series:

Abstract

We address the challenging problem of Natural Language Comprehension beyond plain-text documents by introducing the TILT neural network architecture which simultaneously learns layout information, visual features, and textual semantics. Contrary to previous approaches, we rely on a decoder capable of unifying a variety of problems involving natural language. The layout is represented as an attention bias and complemented with contextualized visual information, while the core of our model is a pretrained encoder-decoder Transformer. Our novel approach achieves state-of-the-art results in extracting information from documents and answering questions which demand layout understanding (DocVQA, CORD, SROIE). At the same time, we simplify the process by employing an end-to-end model.

R. Powalski, L. Borchmann, D. Jurkiewicz, T. Dwojak and M. Pietruszk—Contributed equally.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Expected values have always an exact match in CoNLL, but not elsewhere, e.g., it is the case for 20% WikiReading, 27% Kleister, and 93% of SROIE values.

  2. 2.

    http://www.industrydocuments.ucsf.edu/.

  3. 3.

    Per-category test set scores are available after submission on the competition web page: https://rrc.cvc.uab.es/?ch=17&com=evaluation&task=1.

  4. 4.

    Corrections can be obtained by comparing their two public submissions.

References

  1. Cho, M., Amplayo, R., Hwang, S.W., Park, J.: Adversarial TableQA: attention supervision for question answering on tables. In: PMLR (2018)

    Google Scholar 

  2. Choi, E., et al.: QuAC: question answering in context. In: EMNLP (2018)

    Google Scholar 

  3. Chuang, Y., Liu, C., Lee, H., Lee, L.: SpeechBERT: an audio-and-text jointly learned language model for end-to-end spoken question answering. In: ISCA (2020)

    Google Scholar 

  4. Clark, J.H., et al.: TyDi QA: a benchmark for information-seeking question answering in typologically diverse languages. TACL 8, 454–470 (2020)

    Google Scholar 

  5. Dai, J., Li, Y., He, K., Sun, J.: R-FCN: object detection via region-based fully convolutional networks. In: NeurIPS (2016)

    Google Scholar 

  6. Denk, T.I., Reisswig, C.: BERTgrid: contextualized embedding for 2d document representation and understanding. arXiv preprint (2019)

    Google Scholar 

  7. Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., Smith, N.A.: Fine-tuning pretrained language models: weight initializations, data orders, and early stopping. arXiv preprint (2020)

    Google Scholar 

  8. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., Gardner, M.: DROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs. In: NAACL-HLT (2019)

    Google Scholar 

  9. Dwojak, T., Pietruszka, M., Borchmann, Ł., Chłedowski, J., Graliński, F.: From dataset recycling to multi-property extraction and beyond. In: CoNLL (2020)

    Google Scholar 

  10. Ethayarajh, K.: How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In: EMNLP-IJCNLP (2019)

    Google Scholar 

  11. Garncarek, Ł., et al.: LAMBERT: layout-aware (language) modeling using BERT for information extraction. In: Llads, J. et al. (eds.) ICDAR 2021. LNCS, vol. 12822, pp. 532–547 (2021). Accepted to ICDAR 2021

    Google Scholar 

  12. Guu, K., Lee, K., Tung, Z., Pasupat, P., Chang, M.: Retrieval augmented language model pre-training. In: ICML (2020)

    Google Scholar 

  13. Han, K., et al.: A survey on visual transformer. arXiv preprint (2021)

    Google Scholar 

  14. Harley, A.W., Ufkes, A., Derpanis, K.G.: Evaluation of deep convolutional nets for document image classification and retrieval. In: ICDAR (2015)

    Google Scholar 

  15. Herzig, J., Nowak, P.K., Müller, T., Piccinno, F., Eisenschlos, J.: TaPas: weakly supervised table parsing via pre-training. In: ACL (2020)

    Google Scholar 

  16. Hewlett, D., et al.: WikiReading: a novel large-scale language understanding task over Wikipedia. In: ACL (2016)

    Google Scholar 

  17. Ho, J., Kalchbrenner, N., Weissenborn, D., Salimans, T.: Axial attention in multidimensional transformers. arXiv preprint (2019)

    Google Scholar 

  18. Hong, T., Kim, D., Ji, M., Hwang, W., Nam, D., Park, S.: BROS: a pre-trained language model for understanding texts in document. openreview.net preprint (2021)

    Google Scholar 

  19. Huang, Z., et al.: ICDAR2019 competition on scanned receipt OCR and information extraction. In: ICDAR (2019)

    Google Scholar 

  20. Hwang, W., Yim, J., Park, S., Yang, S., Seo, M.: Spatial dependency parsing for semi-structured document information extraction. arXiv preprint (2020)

    Google Scholar 

  21. Jaume, G., Ekenel, H.K., Thiran, J.P.: FUNSD: a dataset for form understanding in noisy scanned documents. In: ICDAR-OST (2019)

    Google Scholar 

  22. Kafle, K., Price, B.L., Cohen, S., Kanan, C.: DVQA: understanding data visualizations via question answering. In: CVPR (2018)

    Google Scholar 

  23. Kahou, S.E., Michalski, V., Atkinson, A., Kádár, Á., Trischler, A., Bengio, Y.: FigureQA: an annotated figure dataset for visual reasoning. In: ICLR (2018)

    Google Scholar 

  24. Kasai, J., Pappas, N., Peng, H., Cross, J., Smith, N.A.: Deep encoder, shallow decoder: reevaluating the speed-quality tradeoff in machine translation. arXiv preprint (2020)

    Google Scholar 

  25. Keskar, N., McCann, B., Xiong, C., Socher, R.: Unifying question answering and text classification via span extraction. arXiv preprint (2019)

    Google Scholar 

  26. Khashabi, D., et al.: UnifiedQA: crossing format boundaries with a single QA system. In: EMNLP-Findings (2020)

    Google Scholar 

  27. Khot, T., Clark, P., Guerquin, M., Jansen, P., Sabharwal, A.: QASC: a dataset for question answering via sentence composition. In: AAAI (2020)

    Google Scholar 

  28. Kudo, T.: Subword regularization: improving neural network translation models with multiple subword candidates. In: ACL (2018)

    Google Scholar 

  29. Kumar, A., et al.: Ask me anything: dynamic memory networks for natural language processing. In: ICML (2016)

    Google Scholar 

  30. Kwiatkowski, T., et al.: Natural questions: a benchmark for question answering research. TACL 7, 453–466 (2019)

    Article  Google Scholar 

  31. Lai, G., Xie, Q., Liu, H., Yang, Y., Hovy, E.: RACE: large-scale ReAding comprehension dataset from examinations. In: EMNLP (2017)

    Google Scholar 

  32. Le, H., Sahoo, D., Chen, N., Hoi, S.: Multimodal transformer networks for end-to-end video-grounded dialogue systems. In: ACL (2019)

    Google Scholar 

  33. Lee, K.-H., Chen, X., Hua, G., Hu, H., He, X.: Stacked cross attention for image-text matching. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 212–228. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_13

    Chapter  Google Scholar 

  34. Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: ACL (2020)

    Google Scholar 

  35. Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: VisualBERT: a simple and performant baseline for vision and language. arXiv preprint (2019)

    Google Scholar 

  36. Liu, X., Gao, F., Zhang, Q., Zhao, H.: Graph convolution for multimodal information extraction from visually rich documents. In: NAACL-HLT (2019)

    Google Scholar 

  37. Ma, J., Qin, S., Su, L., Li, X., Xiao, L.: Fusion of image-text attention for transformer-based multimodal machine translation. In: IALP (2019)

    Google Scholar 

  38. Mathew, M., Karatzas, D., Jawahar, C.: DocVQA: a dataset for VQA on document images. In: WACV, pp. 2200–2209 (2021)

    Google Scholar 

  39. McCann, B., Keskar, N.S., Xiong, C., Socher, R.: The natural language decathlon: multitask learning as question answering. arXiv preprint (2018)

    Google Scholar 

  40. Palm, R.B., Winther, O., Laws, F.: CloudScan - a configuration-free invoice analysis system using recurrent neural networks. In: ICDAR (2017)

    Google Scholar 

  41. Park, S., et al.: CORD: a consolidated receipt dataset for post-OCR parsing. In: Document Intelligence Workshop at NeurIPS (2019)

    Google Scholar 

  42. Powalski, R., Stanislawek, T.: UniCase - rethinking casing in language models. arXiv prepint (2020)

    Google Scholar 

  43. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. Technical report (2019)

    Google Scholar 

  44. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. JMRL (2020)

    Google Scholar 

  45. Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: EMNLP (2016)

    Google Scholar 

  46. Reddy, S., Chen, D., Manning, C.D.: CoQA: a conversational question answering challenge. TACL 7, 249–266 (2019)

    Google Scholar 

  47. Ren, Y., Liu, J., Tan, X., Zhao, Z., Zhao, S., Liu, T.Y.: A study of non-autoregressive model for sequence generation. In: ACL (2020)

    Google Scholar 

  48. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  49. Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: ACL (2016)

    Google Scholar 

  50. Sidorov, O., Hu, R., Rohrbach, M., Singh, A.: TextCaps: a dataset for image captioning with reading comprehension. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 742–758. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_44

    Chapter  Google Scholar 

  51. Singh, A., et al.: Towards VQA models that can read. In: CVPR (2019)

    Google Scholar 

  52. Stanisławek, T., et al.: Kleister: key information extraction datasets involving long documents with complex layouts. In: Llads, J. et al. (eds.) ICDAR 2021. LNCS, vol. 12822, pp. 564–579 (2021). Accepted to ICDAR 2021

    Google Scholar 

  53. Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J.: VL-BERT: pre-training of generic visual-linguistic representations. In: ICLR (2020)

    Google Scholar 

  54. Vaswani, A., et al.: Attention is all you need. In: NeurIPS (2017)

    Google Scholar 

  55. Xu, Y., et al.: LayoutLMv2: multi-modal pre-training for visually-rich document understanding. arXiv preprint (2020)

    Google Scholar 

  56. Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: LayoutLM: pre-training of text and layout for document image understanding. In: KDD (2020)

    Google Scholar 

  57. Yin, P., Neubig, G., Yih, W.t., Riedel, S.: TaBERT: pretraining for joint understanding of textual and tabular data. In: ACL (2020)

    Google Scholar 

Download references

Acknowledgments

The authors would like to thank Filip Graliński, Tomasz Stanisławek, and Łukasz Garncarek for fruitful discussions regarding the paper and our managing directors at Applica.ai. Moreover, Dawid Jurkiewicz pays due thanks to his son for minding the deadline and generously coming into the world a day after.

The Smart Growth Operational Programme supported this research under project no. POIR.01.01.01-00-0877/19-00 (A universal platform for robotic automation of processes requiring text comprehension, with a unique level of implementation and service automation).

Author information

Authors and Affiliations

  1. Applica.ai, Warsaw, Poland

    Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka & Gabriela Pałka

  2. Poznan University of Technology, Poznań, Poland

    Łukasz Borchmann

  3. Adam Mickiewicz University, Poznań, Poland

    Dawid Jurkiewicz, Tomasz Dwojak & Gabriela Pałka

  4. Jagiellonian University, Cracow, Poland

    Michał Pietruszka

Authors
  1. Rafał Powalski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Łukasz Borchmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dawid Jurkiewicz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tomasz Dwojak
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michał Pietruszka
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gabriela Pałka
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Łukasz Borchmann .

Editor information

Editors and Affiliations

  1. Universitat Autònoma de Barcelona, Barcelona, Spain

    Josep Lladós

  2. Lehigh University, Bethlehem, PA, USA

    Daniel Lopresti

  3. Kyushu University, Fukuoka-shi, Japan

    Seiichi Uchida

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Powalski, R., Borchmann, Ł., Jurkiewicz, D., Dwojak, T., Pietruszka, M., Pałka, G. (2021). Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12822. Springer, Cham. https://doi.org/10.1007/978-3-030-86331-9_47

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86331-9_47

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86330-2

  • Online ISBN: 978-3-030-86331-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation