Safia Meklati
Abstract
This article addresses the general problem of built heritage protection against both deterioration and loss. To continuously monitor and update the structural health status, a crowd-sensing solution based on powerful and automatic deep learning technique is proposed. The aim of this solution is to get rid of the limitations of manual and visual damage detection methods that are costly and time-consuming. Instead, automatic visual inspection for damage detection on walls is efficiently and effectively performed using an embedded Convolutional Neural Network (CNN). This CNN detects the most frequent types of surface damage on wall photos. The study has been conducted in the Kasbah of Algiers where the four following types of damages have been considered: Efflorescence, spall, crack, and mold. The CNN is designed and trained to be integrated into a mobile application for a participatory crowd-sensing solution. The application should be widely and freely deployed, so any user can take a picture of a suspected damaged wall and get an instant and automatic diagnosis through the embedded CNN. In this context, we have chosen MobileNetV2 with a transfer learning approach. A set of real images have been collected and manually annotated and have been used for training, validation, and test. Extensive experiments have been conducted to assess the efficiency and the effectiveness of the proposed solution, using a 5-fold cross-validation procedure. Obtained results show in particular a mean weighted average precision of 0.868 ± 0.00862 (with a 99% of confidence level) and a mean weighted average recall of 0.84 ± 0.00729 (with a 99% of confidence level). To evaluate the performance of MobileNetV2 as a feature extractor, we conducted a comparative study with other small backbones. Further analysis of CNN activation using Grad-Cam has also been done. Obtained results show that our method remains effective even when using a small network and medium- to low-resolution images. MobileNetV2-based CNN size is smaller, and computational cost better, compared to the other CNNs, with similar performance results. Finally, detected surface damages have also been plotted on a geographic map, giving a global view of their distribution.
- [1] . 2003. Analysis of edge-detection techniques for crack identification in bridges. J. Comput. Civil Eng. 17, 4 (2003), 255–263.
DOI: Google Scholar
Cross Ref
- [2] . 2022. Effectively detecting left bundle branch block false defects in myocardial perfusion imaging (MPI) with a convolutional neural network (CNN). Stud. Health Technol. Inform. 289 (2022), 216–219.Google Scholar
- [3] . 2017. Image4Act: Online social media image processing for disaster response. In IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). 601–604.Google Scholar
- [4] . 2021. Transfer learning based hybrid 2D-3D CNN for traffic sign recognition and semantic road detection applied in advanced driver assistance systems. Appl. Intell. 51, 1 (2021), 124–142.Google ScholarDigital Library
- [5] . 2018. Benchmark analysis of representative deep neural network architectures. IEEE Access 6 (2018), 64270–64277.
DOI: Google ScholarCross Ref - [6] . 2010. On over-fitting in model selection and subsequent selection bias in performance evaluation. J. Mach. Learn. Res. 11, 70 (2010), 2079–2107. Retrieved from http://jmlr.org/papers/v11/cawley10a.html.Google ScholarDigital Library
- [7] . 2017. Deep learning-based crack damage detection using convolutional neural networks. Comput.-aid. Civil Infrast. Eng. 32, 5 (2017), 361–378.
DOI: Google ScholarDigital Library - [8] . 2018. Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Comput.-aid. Civil Infrast. Eng. 33, 9 (2018), 731–747.
DOI: Google ScholarDigital Library - [9] . 2017. Transfer Learning and Fine-tuning. Tensorflow. Retrieved from https://www.tensorflow.org/tutorials/images/transfer_learning.Google Scholar
- [10] . 2020. Damage assessment of earthen sites of the Ming Great Wall in Qinghai Province: A comparison between support vector machine (SVM) and BP neural network. J. Comput. Cult. Herit. 13, 2 (
May 2020).DOI: Google ScholarDigital Library - [11] . 2019. Autonomous concrete crack detection using deep fully convolutional neural network. Autom. Construct. 99 (2019), 52–58.
DOI: Google ScholarCross Ref - [12] . 2006. An introduction to ROC analysis. Patt. Recog. Lett. 27, 8 (2006), 861–874.
DOI: Google ScholarDigital Library - [13] . 2020. Machine learning for cultural heritage: A survey. Patt. Recog. Lett. 133 (2020), 102–108.
DOI: Google ScholarCross Ref - [14] . 2011. Estimation of prediction error by using K-fold cross-validation. Statist. Comput. 21, 2 (
1 Apr. 2011), 137–146.DOI: Google ScholarDigital Library - [15] . 2018. A genetic algorithm for convolutional network structure optimization for concrete crack detection. In IEEE Congress on Evolutionary Computation (CEC’18). 1–8.
DOI: Google ScholarDigital Library - [16] . 2015. Deep residual learning for image recognition. CoRR abs/1512.03385 (2015).Google Scholar
- [17] . 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 770–778.
DOI: Google ScholarCross Ref - [18] . 2017. Determination of weathering degree of the Persepolis stone under laboratory and natural conditions using fuzzy inference system. Construct. Build. Mater. 145 (2017), 28–41.
DOI: Google ScholarCross Ref - [19] . 2021. UNESCO World Heritage Centre - State of Conservation (SOC 2021) Kasbah of Algiers (Algeria). UNESCO. Retrieved from https://whc.unesco.org/en/soc/4230/.Google Scholar
- [20] . 2017. MobileNets: Efficient convolutional neural networks for mobile vision applications. CoRR abs/1704.04861 (2017).Google Scholar
- [21] . 2016. Densely connected convolutional networks. CoRR abs/1608.06993 (2016).Google Scholar
- [22] . 2021. Global Multi-label Confusion Matrix. Retrieved from https://github.com/jdariasl/global_multiLabel_confusion_matrix.Google Scholar
- [23] . 2020. Performance of the precise point positioning method along with the development of GPS, GLONASS and Galileo systems. Measurement 164 (2020), 108009.
DOI: Google ScholarCross Ref - [24] . 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, Vol. 25. Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf.Google ScholarDigital Library
- [25] . 2020. Multi-label classifier performance evaluation with confusion matrix. Comput. Sci. Inf. Technol. 10, 8 (2020).Google Scholar
- [26] . 2020. Detection of disaster-affected cultural heritage sites from social media images using deep learning techniques. J. Comput. Cult. Herit. 13, 3 (
Aug. 2020).DOI: Google ScholarDigital Library - [27] . 1997. Face recognition: A convolutional neural-network approach. IEEE Trans. Neural Netw. 8, 1 (1997), 98–113.
DOI: Google ScholarDigital Library - [28] . 1989. Backpropagation applied to handwritten zip code recognition. Neural Computat. 1, 4 (1989), 541–551.
DOI: Google ScholarDigital Library - [29] . 2019. Automatic pixel-level multiple damage detection of concrete structure using fully convolutional network. Comput.-aid. Civil Infrast. Eng. 34, 7 (2019), 616–634.Google ScholarDigital Library
- [30] . 2021. Machine learning techniques for structural health monitoring of heritage buildings: A state-of-the-art review and case studies. J. Cultur. Herit. 47 (2021), 227–245.
DOI: Google ScholarCross Ref - [31] . 2018. WWW’18 workshop on exploitation of social media for emergency relief and preparedness: Chairs’ welcome & organization. In the Web Conference (WWW’18). International World Wide Web Conferences Steering Committee, 1609–1611.
DOI: Google ScholarDigital Library - [32] . 2020. Transfer learning with fine-tuned MobileNetV2 for diabetic retinopathy. In International Conference for Emerging Technology (INCET’20). 1–4.
DOI: Google ScholarCross Ref - [33] . 2021. Non invasive detection of moss and crack in monuments using image processing techniques. J. Amb. Intell. Human. Comput. 12, 5 (
1 May 2021), 5277–5285.DOI: Google ScholarCross Ref - [34] . 2022. Mapbox Layers. Plotly. Retrieved from https://plotly.com/python/mapbox-layers/.Google Scholar
- [35] . 2018. Deterioration assessment of infrastructure using fuzzy logic and image processing algorithm. J. Perform. Construct. Facil. 32, 2 (2018), 04018009.
DOI: Google ScholarCross Ref - [36] . 2009. Cross-validation. Springer US, Boston, MA, 532–538.
DOI: Google ScholarCross Ref - [37] . 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. CoRR abs/1506.01497 (2015).Google Scholar
- [38] . 2014. ImageNet large scale visual recognition challenge. CoRR abs/1409.0575 (2014).Google Scholar
- [39] . 2018. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation. CoRR abs/1801.04381 (2018).Google Scholar
- [40] . 2016. Grad-CAM: Why did you say that? Visual explanations from deep networks via gradient-based localization. CoRR abs/1610.02391 (2016).Google Scholar
- [41] . 2019. A survey on image data augmentation for deep learning. J. Big Data 6, 1 (
6 July 2019), 60.DOI: Google ScholarCross Ref - [42] . 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
- [43] . 2014. Going deeper with convolutions. CoRR abs/1409.4842 (2014).Google Scholar
- [44] . 2015. Rethinking the inception architecture for computer vision. CoRR abs/1512.00567 (2015).Google Scholar
- [45] . 2021. EfficientNetV2: Smaller models and faster training. CoRR abs/2104.00298 (2021).Google Scholar
- [46] . 2021. Keras Documentation: Keras Applications. Keras. Retrieved from https://keras.io/api/applications/.Google Scholar
- [47] . 2018. Damage classification for masonry historic structures using convolutional neural networks based on still images. Comput.-aid. Civil Infrast. Eng. 33, 12 (2018), 1073–1089.
DOI: Google ScholarDigital Library - [48] . 2019. Novel system for rapid investigation and damage detection in cultural heritage conservation based on deep learning. J. Infrast. Syst. 25, 3 (2019), 04019020.
DOI: Google ScholarCross Ref - [49] . 2019. Automatic damage detection of historic masonry buildings based on mobile deep learning. Autom. Construct. 103 (2019), 53–66.
DOI: Google ScholarCross Ref - [50] . 2020. Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning. Comput.-aid. Civil Infrast. Eng. 35, 3 (2020), 277–291.
DOI: Google ScholarDigital Library - [51] . 2019. Fruit image classification based on MobileNetV2 with transfer learning technique. In 3rd International Conference on Computer Science and Application Engineering (CSAE’19). Association for Computing Machinery, New York, NY.
DOI: Google ScholarDigital Library - [52] . 2014. Visualizing and understanding convolutional networks. In Computer Vision – ECCV 2014, , , , and (Eds.). Springer International Publishing, Cham, 818–833.Google Scholar
- [53] . 2016. Incentives for mobile crowd sensing: A survey. IEEE Commun. Surv. Tutor. 18, 1 (2016), 54–67.
DOI: Google ScholarDigital Library - [54] . 2017. Learning transferable architectures for scalable image recognition. CoRR abs/1707.07012 (2017).Google Scholar
- [55] . 2008. Description of the weathering states of building stones by fractal geometry and fuzzy inference system in the Olba ancient city (Southern Turkey). Eng. Geol. 101, 3 (2008), 124–133.
DOI: Google ScholarCross Ref - [56] . 2019. CNN-based statistics and location estimation of missing components in routine inspection of historic buildings. J. Cultur. Herit. 38 (2019), 221–230.
DOI: Google ScholarCross Ref
Index Terms
- Surface Damage Identification for Heritage Site Protection: A Mobile Crowd-sensing Solution Based on Deep Learning
Recommendations
Damage analysis and quantification of RC beams assisted by Damage-T Generative Adversarial Network
AbstractThe application of Artificial Intelligence (AI) is a popular trend to make damage inspection and analysis in structural health monitoring more intelligent and automatic. However, the existing AI-based approaches, especially vision-...
Content Based Image Retrieval System Based on Deep Convolution Neural Network Model by Integrating Three-Fold Geometric Augmentation
AbstractDeep learning models depend on sizeable labelled training samples, and it is a common challenge that affects image retrieval based applications. Applications that are affected by these issues are computer vision and search engines. In the ...
Ore Image Classification Based on Improved CNN
AbstractThe identification of ore deposits is an important technical task in mining and excavation. However, conventional techniques are time-consuming and tedious. Therefore, data augmentation and transfer learning were used in this topic to ...
Comments