Remote sensing techniques for automated marine mammals detection: a review of methods and current challenges

Introduction

Marine mammals currently face various anthropogenic threats such as fishery bycatch, vessel strikes, competition for resources with commercial fisheries, noise pollution, bioaccumulation of pathogens and toxins, harmful algal blooms, and climate change (Merrick, Silber & De Master, 2018). In this context, surveying efforts to improve knowledge of marine mammal species’ distributions and abundance are critical for their conservation. Surveys can be carried out from land, aircraft or ships for cetaceans (Hiby & Hammond, 1989). Pinnipeds have been usually counted from land-based viewing stations, although aerial photographs have also provided a viable alternative (Moore, Forney & Weller, 2018). Boat surveys around rookeries are also commonly performed (e.g., Arias-del Razo et al., 2016; Adame et al., 2017). However, crewed aerial and vessel platforms can bias observations due to the animals’ reactions to them (Würsig et al., 1998; Born Riget, Dietz & Andriashek, 1999; Dawson et al., 2004; Luksenburg & Parsons, 2009; Hashim & Jaaman, 2011). In addition, crewed aerial and boat surveys, as well as ground surveys, can be dangerous to researchers (Sasse, 2003; Hodgson, Kelly & Peel, 2013; Gooday et al., 2018), expensive, and logistically challenging for remote populations far away from an airstrip or a port (Hodgson, Kelly & Peel, 2013; Gooday et al., 2018; Höschle et al., 2021).

In this context, satellite and unoccupied aerial systems (UAS) imagery have surged as complementary tools to count individuals of various taxa of marine mammals (Sirenia, Ursidae, Pinnipedia, and Cetacea: cf. Table 1). These two platforms share the advantage of minimizing danger to researchers. Furthermore, the two platforms can provide still images researchers can revisit and use to cross-validate their counts with other researchers (e.g., Gonçalves, Spitzbart & Lynch, 2020). In particular, very-high-resolution (VHR) satellite imagery (i.e., sub-meter spatial resolution) allows monitoring marine mammals in remote areas (LaRue, Stapleton & Anderson, 2017) without any disturbance to wildlife (LaRue et al., 2011). As a result of a wide geographic coverage, VHR satellite images have the potential to fill knowledge gaps on distribution, abundance, density, and population trends of different marine mammal taxa (cf. Table 1) while supplementing data from field-based surveys and allowing for continued monitoring when in-person surveys are not possible (Höschle et al., 2021). A comparison between satellite and ship survey density estimates showed encouraging results for satellite imagery (i.e., Bamford et al., 2020). On the other hand, despite still requiring fieldwork and offering less area coverage than satellites, UAS allow researchers more freedom in terms of survey timing than satellites and permit coverage in regions of perennial overcast weather (e.g., Goebel et al., 2015). Adame et al. (2017) found their UAS survey to be more accurate than a traditional boat-based survey for the study of a California sea lion (Zalophus californianus) rookery. Furthermore, UAS collect multispectral imagery of higher resolution than satellites, at centimeter (Johnston, 2019) and even sub-centimeter spatial resolutions (Raoult et al., 2020), which is especially relevant for smaller marine mammals like mustelids (e.g., sea otters - Enhydra lutris). Moreover, UAS can provide thermal infrared (IR) imagery at sufficient resolution to detect marine mammals on land (e.g., Seymour et al., 2017), unlike satellite-based thermal IR, which offers too coarse of a resolution (e.g., United States Geological Survey, 2020).

Despite their advantages, satellite and UAS platforms are not panaceas. The cost of satellite imagery still constitutes a limiting factor for many studies (Turner et al., 2015; Höschle et al., 2021). UAS, on the other hand, can be disturbing to animals depending on their noise profile (Pomeroy, O’Connor & Davies, 2015). Furthermore, high-endurance UAS recommended to cover large regions (Raoult et al., 2020) are costly, and their use can be limited based on civil aviation regulations (Fiori et al., 2017). In such cases, satellites remain the most cost-effective solution to study large-bodied animals over large areas (Johnston, 2019). In addition to these issues that depend on costs, regulatory framework, and technological advancements of these platforms, manual counts of animals in remote sensing data are extremely labor and time-intensive for researchers over large areas (Hollings et al., 2018; Höschle et al., 2021). For example, the analysis of 425 km² (still a relatively small area in terms of marine mammal distribution) took 24 h to be analyzed with three replicates (eight hours per person) for the work of Thums et al. (2018), corresponding to 1.13 min per km². Furthermore, Cubaynes et al. (2019) reported three hours and 20 min to scan 100 km² at a 1:1,500 m. scale (2 min per km²), and Charry et al. (2021) reported an even slower rate of approximately 2.5 min per km² (at a 1:536 scale). While satellite images have allowed continental-scale abundance and distribution studies in remote locations (e.g., LaRue et al., 2019; LaRue et al., 2020; LaRue et al., 2021), that work required crowdsourcing: LaRue et al. (2020) enlisted more than 325,000 volunters to analyze 268,611 km² of images while a previous effort (LaRue et al., 2019) used more than 5,000 volunteers for the Ross Sea region in Antarctica. These broad-scale studies remain scarce for marine mammals: most work based on satellite images is performed over smaller areas such as individual islands (e.g., LaRue et al., 2015; LaRue & Stapleton, 2018).

Automation of image analysis reduces the level of effort and time needed, especially for large-scale projects (Hollings et al., 2018; Charry et al., 2021; Höschle et al., 2021). Even semi-automation in Thums et al. (2018) shortened 24 person-hours to 34 min to run the algorithm and an additional 20 min to revise sub-sections with whale detections and discard those that were false (a reduction of more than 96% of the analysis time). Automation could also help with errors caused by human fatigue from reviewing imagery (Hodgson, Kelly & Peel, 2013). As such, automation could unleash the full potential of VHR satellite images to cover more extensive study areas, which is ultimately a capability that distinguishes the use of satellite imagery from crewed surveys. Automating animal detection in UAS imagery has also been suggested to provide better survey data (Hodgson, Peel & Kelly, 2017). Just as for satellite imagery, automation would also reduce the labor and time necessary to count animals from UAS imagery (Linchant et al., 2015). Technologies for automated count of animals have emerged for areas difficult to access, in which ground counts are difficult, or for animals that are at low densities over large areas (Hollings et al., 2018). Marine mammals generally fit all these conditions, given their aquatic habitats and extended ranges, making them suitable candidates for the application of automated counts. However, automated detection in UAS imagery has been more often applied to terrestrial mammals and birds than to marine mammals (Corcoran et al., 2021), and Hollings et al. (2018) report a similar pattern for automated analysis of remote sensing imagery in general.

Automated analysis of remote sensing imagery can be divided into pixel or object-based methods, depending on the unit of analysis (Blaschke et al., 2014; Wang, Shao & Yue, 2019). While pixel-based methods (e.g., unsupervised or supervised classification and thresholding) classify individual pixels into animal or background classes based on spectral information or texture, object-based methods classify groups of homogeneous pixels into the different classes, which allows for the use of other variables, such as size and shape (Blaschke et al., 2014; Wang, Shao & Yue, 2019). This contribution reviews the use of different methods of automated analysis of VHR satellite and UAS imagery for marine mammal detection, which is a current gap as previously-published reviews are dedicated to wildlife in general, either to their automated detection in UAS (i.e., Corcoran et al., 2021) or manual and automatic detection in UAS, satellite and aerial imagery (i.e., Hollings et al., 2018; Wang, Shao & Yue, 2019). The detection of marine mammals in their natural habitats presents different challenges that warrant their own review. The analysis we present also adds to contributions that focused on using satellite (LaRue, Stapleton & Anderson, 2017; Höschle et al., 2021) and UAS (Koski, Abgrall & Yazvenko, 2010; Smith et al., 2016; Fiori et al., 2017; Johnston, 2019; Schofield et al., 2019; Raoult et al., 2020) imagery to study marine wildlife separately. This review aims to inform the multidisciplinary community of remote sensing experts, biologists, ecologists, and managers working on marine mammal research and conservation as to which automated methods and imagery type show more promise, but also which taxa have been overlooked, practical alternatives to ensure time-saving and accurate counts, and steps necessary to progress from individual studies to the regional-scale initiatives that conservation needs. Our analysis is also summarized in Table 1, which improves upon the work of Hollings et al. (2018) and Cubaynes (2019) by the application of uniform metrics for evaluating detection accuracy that allow comparing the results of existing studies directly and for a more straightforward assessment of over- and undercounts in a conservation context.

Discussion

The current threats marine mammals face justify the need for their populations to be monitored through spatially extensive and frequent surveys for abundance and distribution studies. Satellite and UAS imagery can help address these challenges and complement crewed surveys. Satellite images can provide extensive coverage, enabling even continental-scale counts to use for abundance and distribution estimations in remote locations (e.g., LaRue et al., 2019; LaRue et al., 2020; LaRue et al., 2021), making sampling efforts more globally even. Although UAS provide less spatial coverage than satellite images, they can allow more control over survey timing and surveying in overcast conditions, as well as higher spatial resolution, which can be of use for smaller marine mammals like sea otters (Enhydra lutris) or dolphins of the Cephalorhynchus genus. Nevertheless, the manual analysis of satellite and UAS imagery is time- and effort-intensive, for which automation can help (Linchant et al., 2015; Hollings et al., 2018; Charry et al., 2021; Höschle et al., 2021). While Thums et al. (2018) provided a comparative measure of time saved by their workflow, we recommend future automated marine mammal studies do the same, i.e., that they evaluate how efficient different automated methods are in saving time and effort compared to a manual approach.

We also acknowledge the current challenges of automated approaches in matching the animals present in the imagery. In the studies we analyzed, automation efforts thus far have shown considerable deviation of their detections over the animals manually counted, in particular when it came to false positives (cf. Table 1). This is especially concerning in a conservation assessment context, as such an error could lead to the lifting of protection measures when they should stay in place, and therefore calls for future research to improve detection methods. Until automated methods are better, studies on marine mammal detection should compromise with semi-automated approaches that involve a user review (Höschle et al., 2021). However, one positive aspect is that even if a manual review is needed, a semi-automated method that yields image sub-sections or thumbnails that need to be revised manually afterwards can still shorten the overall revision time, as we have seen from Thums et al. (2018). Dujon et al. (2021) also recommend semi-automation in the sense of leaving full automation for situations of animals that show optimal conditions for such analysis, and doing manual detections in more challenging settings. Furthermore, the authors recognized that while their CNN could be used in different species even within a single site, a sole algorithm would not be able to capture all components of different species optimally as a result of morphology, spacing behavior, and habitat (Dujon et al., 2021).

Here, we have reviewed the automated approaches for detecting marine mammals through satellite and UAS images. We observed that the automated methods reliant only on spectral information did not show high-performance (e.g., Fretwell, Staniland & Forcada, 2014; LaRue et al., 2015), and spectral contrast between baleen whales and their surroundings were found to be insufficient for automation, highlighting the need for other methods that can accommodate more information, such as deep learning and OBIA (Cubaynes et al., 2019). Deep learning showed high performance in some studies (i.e., Guirado et al., 2019; Gray et al., 2019) and even allowed for distinguishing between whale species (i.e., Gray et al., 2019), although this has not yet been achieved in satellite images (Höschle et al., 2021). Nevertheless, not all deep learning attempts were successful, and some attempts required hundreds (e.g., Gray et al., 2019) or thousands (e.g., Guirado et al., 2019) of images for training. An extra complication is that labeled satellite images are currently limited, highlighting the need for shared training datasets (Höschle et al., 2021). Overall, CNN shows promise, and future research should continue to improve on the studies already conducted. For instance, CNNs only successful attempts thus far have been in whales—although the contrary is not true—for which further studies are due to see if the method’s results in pinnipeds can be improved, and to evaluate the method in other smaller marine mammal taxa. The number of studies using CNN is still low, and more studies need to be undertaken to identify which components of workflows improve accuracy. For example, the use of more confounding classes seemed useful for Guirado et al. (2019) but was not a guarantee of success for Gonçalves, Spitzbart & Lynch (2020).

We agree with Hollings et al. (2018) in that less costly images and software are necessary to drive the field forward. The use of free platforms for image analysis, such as Google Earth Engine, could provide a solution by also providing greater computational power to researchers. We recommend the development of semi-automated methods that can be applied to the simultaneous detection of multiple species sharing a common habitat. Therefore, free, regional, or even continental initiatives that combine training data for recognition of different species, through the cooperation among different research groups, government agencies or nonprofit organizations, are what is now necessary to provide a useful tool for conservation managers to obtain counts for abundances and distribution studies. Moreover, with readily available and accessible satellite image coverage, Abileah (2001) and Cubaynes et al. (2019) recommendations to use satellite imagery to study migrations could be finally implemented at a large scale, and object-based methods could provide exclusive applications, such as the detection of swimming directions of different individuals. This could complement the use of tagging devices and lead to mapping migration routes for policy measures aiming to reduce collisions and bycatch. Furthermore, accessible satellite image coverage combined with the speed of automation could help improve distribution estimations, discover unknown feeding and calving grounds, and keep track of distribution shifts due to climate change. Moreover, it is worth considering baleen whales have been the focus of most automated studies, and although some automated studies were applied to pinnipeds, sirenians, and polar bears, more studies are needed to evaluate different platforms, automation methods, or spectral bands for these other taxa. Automated studies on toothed whales and mustelids are also due. Here we focused on automated detection of living marine mammal individuals in VHR satellite and UAS imagery, but it is noteworthy that other promising satellite and UAS image analysis applications (automated and manual) exist and focus for example on detecting marine mammal groups (e.g., Burn & Cody, 2005; Fischbach & Douglas, 2021), traces and holes (e.g., Platonov, Mordvintsev & Rozhnov, 2013), carcasses (e.g., Fretwell et al., 2019; Clarke et al., 2021), and on comparing detection success from imagery of different spatial resolutions (e.g., Fischbach & Douglas, 2021; Corrêa et al., 2022).

Regarding new promising directions to explore, OBIA has been recommended for automated analysis (Cubaynes et al., 2019), but thus far pixel-based methods have been more thoroughly explored and the OBIA studies or the studies using object features have shown mixed results (e.g., Mejias Alvarez et al., 2013; Maire et al., 2013; Seymour et al., 2017; Thums et al., 2018; Cubaynes, 2019; Zinglersen et al., 2020). Moreover, UAS offer data at sufficient resolution in the thermal IR, which has proven useful in distinguishing between pinniped pups and adults and counting individuals in aggregations (i.e., Seymour et al., 2017). The use of multispectral imagery has been prevalent among studies thus far, and we recommend further exploring the use of thermal IR as a solution to distinguish animals from similarly-colored surroundings, an issue spanning different taxa and surfaces (e.g., LaRue et al., 2015; Cubaynes et al., 2019; LaRue et al., 2020), or even to study marine mammals nighttime behavior, or during the polar night (Boehme et al., 2016). Furthermore, we decided to use the FNR (Eq. (3)), the FPDR (Eq. (4)) and the automated count deviation (Eq. (5)) proposed as a more straightforward means of assessing how reliable an automated method would be if adopted in conservation planning, and what the impact of its errors would be over counts later used in abundance estimations.

Marine mammal abundance and distribution studies can be well served from a combination of different surveying methods. For instance, UAS and satellite images, airplane and vessel surveys, to detect animals in the surface; passive acoustic monitoring, to detect submerged animals’ vocalizations (e.g., Davis et al., 2017; Stanistreet et al., 2018); and satellite tracking, to obtain detailed movement patterns (e.g., Horton et al., 2017; Horton et al., 2020). In the last decade, studies using satellite and UAS imagery for marine mammal surveys have grown considerably in number, but lag behind the number of automated studies on other taxa. This may occur because of the logistical complexity of conducting UAS surveys at sea as opposed to on land (e.g., launching and landing sites, weather conditions), as well as the relative scarcity of archival satellite imagery over sea compared to terrestrial habitats, leaving researchers only with more expensive tasking options. Automated methods have the potential to complement airplane and vessel marine mammal surveys, helping overcome old limitations, such as higher coverage in remote regions. However, to do that, satellite imagery needs to become more available and affordable further away from the coast, and high-endurance UAS need to become more affordable too, while their regulation needs to be flexibilized. In this context, joint initiatives for further research are necessary to not only to exchange imagery, but also to develop automated methods that both save researchers time and provide accurate counts, while being versatile enough to work in different settings and distinguish coexistent species.

Supplemental Information