DeepSmoke: Deep learning model for smoke detection and segmentation in outdoor environments
Introduction
Disaster management is a wide domain of research to which many researchers have contributed in recent years (Finney, 2020, Muhammad et al., 2020) because of its direct relevance to human lives and properties. There are several categories of disasters (Lu et al., 2019, Muhammad, Hussain, Tanveer, Sannino, & de Albuquerque, 2019), including e.g. flood and fire, which need to be detected and monitored at their early stages to allow preventive actions. Among these disasters, fire is the most dangerous and can bring enormous damages (Bilbao et al., 2015, Cui, 2020). Thus, its automatic detection in IoT environments plays a vital role in the early handling of fire (Abbas, Zhang, Taherkordi, & Skeie, 2018). The identification of smoke is a primary sign of fire and its early detection is an effective way of averting damages caused by fire (Tian, Li, Wang, & Ogunbona, 2014). To prevent damages caused by fire disaster, several traditional and vision sensor-based fire and smoke detection methods (Muhammad et al., 2018, Muhammad et al., 2020) have been proposed. Among these methods, vision-based smoke detection systems have attracted much attention within the research community.
Towards smoke detection through vision sensors, the employed methods can be divided into two major categories: traditional and deep learning-based approaches. The mainstream methods belonging to the first category use different features such as color, shape, and texture to identify smoke regions. These features identify smoke without involving any learning process. Furthermore, some of these methods use low-level features and passes them to different classifiers or clustering techniques for smoke prediction. These methods have several limitations including high False Alarm Rate (FAR), limited accuracy, and lower detection range.
In contrast to the traditional approaches for smoke detection, the deep learning-based methods utilize learned features for the identification and segmentation of smoke patterns. These methods are reinforced by convolutions, pooling, and fully connected layers for learning visual notions. Several CNN-based smoke detection methods exist in the literature, which employ different models for both smoke detection and segmentation. Unfortunately, the huge model size and higher computational complexity of these modeling choices restrict their usage in real-time IoT scenarios (Liu et al., 2019, Muhammad, Hussain, Tanveer, Sannino, & de Albuquerque, 2019).
To tackle these issues effectively, we pose this novel research for smoke detection and segmentation using efficient CNNs that are functional in real-world IoT environments. Specifically, we propose an efficient smoke detection and semantic segmentation method for outdoor clear and hazy environments. The proposed framework comprises two modules: the first one is smoke detection and the second one is semantic segmentation of smoky regions. In the first module, we follow a classification strategy for the identification of smoke inside video frames. To this purpose, we use a smoke detection dataset in a hazy environment, consisting of four different classes: “smoke”, “non-smoke”, “smoke with fog”, and “non-smoke with fog”. For classification, we fine-tune a pre-trained EfficientNet architecture (Tan & Le, 2019), that is highly efficient and precise when compared to state-of-the-art CNN models. After the successful detection of smoke, the next step is to segment the smoke regions from the detected frames. We tackle this problem through a recent semantic segmentation CNN model, DeepLabv3+ (Chen, Zhu, Papandreou, Schroff, & Adam, 2018). The latter has efficient encoder and decoder layers, followed by a pixel-wise classifier for localization of smoke regions that are useful for disaster management. The main contributions of this paper can be summarized as follows:
- •
We present “DeepSmoke” (a deep learning framework for smoke detection and segmentation), by incorporating two essential modules for intelligent disaster management, smoke detection, and segmentation. The key ingredient of our framework is its high-level of adaptability to clear and hazy surveillance environments for smoke detection and localization.
- •
Smoke segmentation plays a key role in fire scene contextual analysis and its respective reporting. Its relevant literature lacks publicly available challenging and real-world datasets, which restrict the contributions from researchers to the mentioned domain. To handle this issue, we created our own smoke segmentation dataset by manually labelling the smoke regions. The dataset is made publicly available for research community (https://github.com/salmank255/deepsmoke).
- •
Two sets of experiments are carried out to show the effectiveness of DeepSmoke: a) an evaluation of different CNN models, and b) a comparison with existing smoke detection and segmentation techniques from different perspectives i.e., FAR, accuracy, and mean Intersection over Union. Finally, we advocate for the usage of EfficientNet for smoke detection and DeepLabv3+ for segmentation as the optimum CNN models for the said problem.
- •
Unlike the recently devised smoke segmentation technique (Yuan et al., 2019) that relies only on localization without focusing on smoke detection, we tackle both tasks using efficient and state-of-the-art CNN models. Automatic smoke detection followed by segmentation can provide a complete functional system with the advantage of providing instant alert generation in IoT networks and ensuring reduced FAR.
The rest of this paper is organized as follows. Section 2 explains the detailed literature of smoke detection and segmentation based on both traditional and deep learning-based methods. Section 3 provides the complete details of our proposed framework for smoke detection and segmentation in both indoor and outdoor environments. Extensive experiments of our framework on both surveillance and wildfires data are explained in Section 4. Finally, Section 5 concludes the paper by concentrating on the main building blocks of the proposed method with future research directions.
Section snippets
Related works
In this section, we thoroughly review existing smoke detection and segmentation techniques. The overall section is divided into traditional (Section 2.1) and deep learning-based (Section 2.2) smoke detection techniques.
Proposed DeepSmoke framework
In order to detect fire at its early stages, we propose a novel lightweight smoke detection and segmentation framework coined as DeepSmoke. The overall pipeline of our method is divided into two modules. The first module performs smoke detection using a lightweight CNN model, followed by a second module addressing semantic smoke segmentation. In the following subsections we delve into the technical details of each of these modules:
Experiments
This section describes the experimental details including dataset and simulation environment used for the testing. Next, a comparison of our model with the other state-of-the-art deep learning architectures is presented. Finally, our “DeepSmoke” is extensively compared with recent smoke detection and segmentation methods from various perspectives.
Conclusion
Mainstream vision-based smoke detection techniques are based on low-level features and statistical learning, limited to indoor or outdoor scenarios without any focus on hazy environments. In this paper we fine-tuned an EfficientNet architecture for detection of smoke, non-smoke, smoke with fog, and non-smoke with fog. Our system significantly improved the False Alarm Rate by reporting a reduction of 1.98%, and by boosting the accuracy to 98.18% when compared to the state-of-the-art models and
CRediT authorship contribution statement
Salman Khan: Conceptualization, Methodology, Software, Writing - original draft. Khan Muhammad: Conceptualization, Methodology, Software, Writing - original draft, Formal analysis, Investigation, Validation, Writing - review & editing, Supervision. Tanveer Hussain: Methodology, Software, Writing - original draft. Javier Del Ser: Methodology, Investigation, Validation, Writing - review & editing. Fabio Cuzzolin: Investigation, Validation, Writing - review & editing, Supervision. Siddhartha
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgement
The work of Salman Khan and Fabio Cuzzolin has received funding from the European Union’s Horizon 2020 research and innovation programme, under grant agreement No. 964505 (E-pi). J. Del Ser acknowledges funding support from the Basque Government through the ELKARTEK program (3KIA project, KK-2020/00049) and the consolidated research group MATHMODE (ref. T1294-19).
References (54)
- et al.
Semantic object classes in video: A high-definition ground truth database
Pattern Recognition Letters
(2009) Deployment and integration of smart sensors with IoT devices detecting fire disasters in huge forest environment
Computer Communications
(2020)- et al.
Estimating wildfire-generated ozone over North America using ozonesonde profiles and a differential back trajectory technique
Atmospheric Environment: X
(2020) - et al.
An effective four-stage smoke-detection algorithm using video images for early fire-alarm systems
Fire Safety Journal
(2011) - et al.
Dynamic texture based smoke detection using Surfacelet transform and HMT model
Fire Safety Journal
(2015) A fast accumulative motion orientation model based on integral image for video smoke detection
Pattern Recognition Letters
(2008)- et al.
Deep smoke segmentation
Neurocomputing
(2019) - et al.
Mobile edge computing: A survey
IEEE Internet of Things Journal
(2018) - et al.
Smoke detection using temporal HOGHOF descriptors and energy colour statistics from video
Paper presented at the International workshop on multi-sensor systems and networks for fire detection and management
(2012) - et al.
Segnet: A deep convolutional encoder-decoder architecture for image segmentation
IEEE Transactions on Pattern Analysis and Machine Intelligence
(2017)