DeepSmoke: Deep learning model for smoke detection and segmentation in outdoor environments

https://doi.org/10.1016/j.eswa.2021.115125Get rights and content

Highlights

  • Smoke detection and localization in both clear and hazy outdoor environments.

  • Using a lightweight CNN architecture called EfficientNet for smoke detection.

  • Employing DeepLabv3+ semantic segmentation architecture for smoke localization.

  • Pixel-wise annotation of a new benchmark dataset for smoke semantic segmentation.

  • Outperformed existing smoke detection and segmentation methods.

Abstract

Fire disaster throughout the globe causes social, environmental, and economical damage, making its early detection and instant reporting essential for saving human lives and properties. Smoke detection plays a key role in early fire detection but majority of the existing methods are limited to either indoor or outdoor surveillance environments, with poor performance for hazy scenarios. In this paper, we present a Convolutional Neural Network (CNN)-based smoke detection and segmentation framework for both clear and hazy environments. Unlike existing methods, we employ an efficient CNN architecture, termed EfficientNet, for smoke detection with better accuracy. We also segment the smoke regions using DeepLabv3+, which is supported by effective encoders and decoders along with a pixel-wise classifier for optimum localization. Our smoke detection results evince a noticeable gain up to 3% in accuracy and a decrease of 0.46% in False Alarm Rate (FAR), while segmentation reports a significant increase of 2% and 1% in global accuracy and mean Intersection over Union (IoU) scores, respectively. This makes our method a best fit for smoke detection and segmentation in real-world surveillance settings.

Introduction

Disaster management is a wide domain of research to which many researchers have contributed in recent years (Finney, 2020, Muhammad et al., 2020) because of its direct relevance to human lives and properties. There are several categories of disasters (Lu et al., 2019, Muhammad, Hussain, Tanveer, Sannino, & de Albuquerque, 2019), including e.g. flood and fire, which need to be detected and monitored at their early stages to allow preventive actions. Among these disasters, fire is the most dangerous and can bring enormous damages (Bilbao et al., 2015, Cui, 2020). Thus, its automatic detection in IoT environments plays a vital role in the early handling of fire (Abbas, Zhang, Taherkordi, & Skeie, 2018). The identification of smoke is a primary sign of fire and its early detection is an effective way of averting damages caused by fire (Tian, Li, Wang, & Ogunbona, 2014). To prevent damages caused by fire disaster, several traditional and vision sensor-based fire and smoke detection methods (Muhammad et al., 2018, Muhammad et al., 2020) have been proposed. Among these methods, vision-based smoke detection systems have attracted much attention within the research community.

Towards smoke detection through vision sensors, the employed methods can be divided into two major categories: traditional and deep learning-based approaches. The mainstream methods belonging to the first category use different features such as color, shape, and texture to identify smoke regions. These features identify smoke without involving any learning process. Furthermore, some of these methods use low-level features and passes them to different classifiers or clustering techniques for smoke prediction. These methods have several limitations including high False Alarm Rate (FAR), limited accuracy, and lower detection range.

In contrast to the traditional approaches for smoke detection, the deep learning-based methods utilize learned features for the identification and segmentation of smoke patterns. These methods are reinforced by convolutions, pooling, and fully connected layers for learning visual notions. Several CNN-based smoke detection methods exist in the literature, which employ different models for both smoke detection and segmentation. Unfortunately, the huge model size and higher computational complexity of these modeling choices restrict their usage in real-time IoT scenarios (Liu et al., 2019, Muhammad, Hussain, Tanveer, Sannino, & de Albuquerque, 2019).

To tackle these issues effectively, we pose this novel research for smoke detection and segmentation using efficient CNNs that are functional in real-world IoT environments. Specifically, we propose an efficient smoke detection and semantic segmentation method for outdoor clear and hazy environments. The proposed framework comprises two modules: the first one is smoke detection and the second one is semantic segmentation of smoky regions. In the first module, we follow a classification strategy for the identification of smoke inside video frames. To this purpose, we use a smoke detection dataset in a hazy environment, consisting of four different classes: “smoke”, “non-smoke”, “smoke with fog”, and “non-smoke with fog”. For classification, we fine-tune a pre-trained EfficientNet architecture (Tan & Le, 2019), that is highly efficient and precise when compared to state-of-the-art CNN models. After the successful detection of smoke, the next step is to segment the smoke regions from the detected frames. We tackle this problem through a recent semantic segmentation CNN model, DeepLabv3+ (Chen, Zhu, Papandreou, Schroff, & Adam, 2018). The latter has efficient encoder and decoder layers, followed by a pixel-wise classifier for localization of smoke regions that are useful for disaster management. The main contributions of this paper can be summarized as follows:

  • We present “DeepSmoke” (a deep learning framework for smoke detection and segmentation), by incorporating two essential modules for intelligent disaster management, smoke detection, and segmentation. The key ingredient of our framework is its high-level of adaptability to clear and hazy surveillance environments for smoke detection and localization.

  • Smoke segmentation plays a key role in fire scene contextual analysis and its respective reporting. Its relevant literature lacks publicly available challenging and real-world datasets, which restrict the contributions from researchers to the mentioned domain. To handle this issue, we created our own smoke segmentation dataset by manually labelling the smoke regions. The dataset is made publicly available for research community (https://github.com/salmank255/deepsmoke).

  • Two sets of experiments are carried out to show the effectiveness of DeepSmoke: a) an evaluation of different CNN models, and b) a comparison with existing smoke detection and segmentation techniques from different perspectives i.e., FAR, accuracy, and mean Intersection over Union. Finally, we advocate for the usage of EfficientNet for smoke detection and DeepLabv3+ for segmentation as the optimum CNN models for the said problem.

  • Unlike the recently devised smoke segmentation technique (Yuan et al., 2019) that relies only on localization without focusing on smoke detection, we tackle both tasks using efficient and state-of-the-art CNN models. Automatic smoke detection followed by segmentation can provide a complete functional system with the advantage of providing instant alert generation in IoT networks and ensuring reduced FAR.

The rest of this paper is organized as follows. Section 2 explains the detailed literature of smoke detection and segmentation based on both traditional and deep learning-based methods. Section 3 provides the complete details of our proposed framework for smoke detection and segmentation in both indoor and outdoor environments. Extensive experiments of our framework on both surveillance and wildfires data are explained in Section 4. Finally, Section 5 concludes the paper by concentrating on the main building blocks of the proposed method with future research directions.

Section snippets

Related works

In this section, we thoroughly review existing smoke detection and segmentation techniques. The overall section is divided into traditional (Section 2.1) and deep learning-based (Section 2.2) smoke detection techniques.

Proposed DeepSmoke framework

In order to detect fire at its early stages, we propose a novel lightweight smoke detection and segmentation framework coined as DeepSmoke. The overall pipeline of our method is divided into two modules. The first module performs smoke detection using a lightweight CNN model, followed by a second module addressing semantic smoke segmentation. In the following subsections we delve into the technical details of each of these modules:

Experiments

This section describes the experimental details including dataset and simulation environment used for the testing. Next, a comparison of our model with the other state-of-the-art deep learning architectures is presented. Finally, our “DeepSmoke” is extensively compared with recent smoke detection and segmentation methods from various perspectives.

Conclusion

Mainstream vision-based smoke detection techniques are based on low-level features and statistical learning, limited to indoor or outdoor scenarios without any focus on hazy environments. In this paper we fine-tuned an EfficientNet architecture for detection of smoke, non-smoke, smoke with fog, and non-smoke with fog. Our system significantly improved the False Alarm Rate by reporting a reduction of 1.98%, and by boosting the accuracy to 98.18% when compared to the state-of-the-art models and

CRediT authorship contribution statement

Salman Khan: Conceptualization, Methodology, Software, Writing - original draft. Khan Muhammad: Conceptualization, Methodology, Software, Writing - original draft, Formal analysis, Investigation, Validation, Writing - review & editing, Supervision. Tanveer Hussain: Methodology, Software, Writing - original draft. Javier Del Ser: Methodology, Investigation, Validation, Writing - review & editing. Fabio Cuzzolin: Investigation, Validation, Writing - review & editing, Supervision. Siddhartha

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

The work of Salman Khan and Fabio Cuzzolin has received funding from the European Union’s Horizon 2020 research and innovation programme, under grant agreement No. 964505 (E-pi). J. Del Ser acknowledges funding support from the Basque Government through the ELKARTEK program (3KIA project, KK-2020/00049) and the consolidated research group MATHMODE (ref. T1294-19).

References (54)

  • P. Barmpoutis et al.

    Smoke detection using spatio-temporal analysis, motion modeling and dynamic texture recognition

    Paper presented at the Signal Processing Conference (EUSIPCO), 2014 Proceedings of the 22nd European

    (2014)
  • M.N. Bilbao et al.

    On the application of multi-objective harmony search heuristics to the predictive deployment of firefighting aircrafts: A realistic case study

    International Journal of Bio-Inspired Computation

    (2015)
  • J. Chen et al.

    Wavelet based smoke detection method with RGB Contrast-image and shape constrain

    Paper presented at the 2013 Visual Communications and Image Processing (VCIP)

    (2013)
  • L.-C. Chen et al.

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2018)
  • L.-C. Chen et al.

    Encoder-decoder with atrous separable convolution for semantic image segmentation

    Paper presented at the Proceedings of the European conference on computer vision (ECCV)

    (2018)
  • M. Cordts et al.

    The cityscapes dataset for semantic urban scene understanding

    Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition

    (2016)
  • K. Dimitropoulos et al.

    Higher order linear dynamical systems for smoke detection in video surveillance applications

    IEEE Transactions on Circuits and Systems for Video Technology

    (2017)
  • M. Everingham et al.

    The pascal visual object classes challenge: A retrospective

    International Journal of Computer Vision

    (2015)
  • A. Filonenko et al.

    Fast smoke detection for video surveillance using CUDA

    IEEE Transactions on Industrial Informatics

    (2018)
  • M.A. Finney

    The wildland fire system and challenges for engineering

    Fire Safety Journal

    (2020)
  • K. He et al.

    Deep residual learning for image recognition

    Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition

    (2016)
  • J. Hu et al.

    Squeeze-and-excitation networks

    Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition

    (2018)
  • Y. Hu et al.

    Real-time video fire smoke detection by utilizing spatial-temporal ConvNet features

    Multimedia Tools and Applications

    (2018)
  • S. Khan et al.

    Energy-efficient deep CNN for smoke detection in Foggy IoT environment

    IEEE Internet of Things Journal

    (2019)
  • A. Krizhevsky et al.

    Imagenet classification with deep convolutional neural networks

    Paper presented at the Advances in neural information processing systems

    (2012)
  • G. Lin et al.

    Smoke detection on video sequences using 3D convolutional neural networks

    Fire Technology

    (2019)
  • J. Liu et al.

    ERP: Edge Resource Pooling for Data Stream Mobile Computing

    IEEE Internet of Things Journal

    (2019)
  • Cited by (0)

    View full text