Underwater Detection: A Brief Survey and a New Multitask Dataset

1. Introduction

Marine resources are extremely valuable to humans. Underwater detection has broad applications in many areas, such as oceanography, marine navigation, and fish farming [1]. However, the complexities of underwater environments and ecosystems present formidable obstacles to effective underwater detection. For example, underwater environments exhibit considerable variability in lighting conditions, depending on factors such as depth and quality [2−3]. Intricate and multifaceted shapes characterize diverse organisms. Certain organisms also evolved a camouflage mechanism. Furthermore, these organisms often face obfuscation or disruption due to the integration of sand and gravel [4]. Therefore, it calls for the integration of related technologies in computer vision and artificial intelligence to address these challenges in underwater detection.

In this context, Object Detection (OD) [5−7] and Salient Object Detection (SOD) [8−10] are crucial to understanding and analyzing underwater scenes. OD is designed to detect and classify objects accurately while providing their precise spatial locations. SOD focuses on the localization and segmentation of salient objects that conform to the human visual system (HVS) [11]. Several studies and surveys have been conducted in the fields of underwater OD and SOD, highlighting the significance of these tasks in underwater image analysis [12−19].

Our work begins with a brief overview of recent breakthroughs and advances within the OD and SOD fields. Different from existing surveys on these two topics [5−10, 12−14, 20−21], we classify OD and SOD models into two primary categories: (i) Convolutional Neural Networks (CNNs) [22] based approaches and (ii) Transformer [23] based approaches. Furthermore, we delve into the unique challenges posed by underwater detection tasks, as well as the strengths and weaknesses of these approaches.

A comprehensive dataset entitled Underwater Object Multitask (UMOT) is also proposed, which contains various underwater scenes with more than 7K instances encompassing three organism types. UOMT provides COCO dataset [24] format labels for OD and binary segmentation masks for SOD, as illustrated in Figure 1. Evaluations of state-of-the-art OD and SOD models are conducted on the UMOT dataset.

Figure 1. Examples in the proposed Underwater Object Multitask (UOMT) dataset: (a) Original images; (b) Annotations for Object Detection; (c) Annotations for Salient Object Detection.

The rest of this article is presented accordingly. Section II briefly reviews OD methods. Section III briefly reviews SOD methods. Section IV explains underwater OD and SOD challenges. Section V describes the evaluation metrics for OD and SOD. Section VI describes the proposed UOMT dataset. Section VII presents benchmarks for OD and SOD for the UOMT dataset through quantitative and qualitative experiments. Section VIII discusses the future development of underwater OD and SOD. Section IX summarizes the main points of this work.

2. Review of Object Detection (OD)

Object detection (OD) endeavors to accurately detect and classify objects, denoting their positions by rectangular boxes [25]. Object detection encompasses various applications, including face detection, instance segmentation, autonomous driving, surveillance systems, and sports analytics [5].

In recent years, many object detection surveys have been conducted. In 2019, Jiao et al. [5] conducted a comparative analysis of different deep learning-based OD methods. It also introduced some commonly used public datasets, analyzed their characteristics, and highlighted their strengths and weaknesses. In 2020, Wu et al. [6] concluded techniques useful in OD, such as attention-based models [26−28], end-to-end models [29−30], depth separable convolutions [31−32], etc. This survey also presented challenges and requirements for practical applications, including detecting small objects, object tracking, real-time performance, and other related topics. Li et al. [7] extensively explored various aspects of OD in images captured by optical sensors or satellites. The survey also proposed an optical remote sensing image dataset with benchmarks. Padilla et al. [12] provided a detailed and comprehensive exposition of the commonly used performance evaluation metrics in object detection methods. This survey provided details on the principles, advantages, and disadvantages of different indicators and their applicability to different application scenarios. In 2022, Cheng et al. [13] provided a comprehensive study and summary of the problem of small object detection. In 2023, Zou et al. [14] systematically reviewed the development of OD over the past 20 years, offering valuable insights into the field of SOD.

Our review of OD categorizes its methods into CNN-based and Transformer-based models. Table 1 lists the models discussed in this section.

2.1. CNN-based Object Detection Models

Convolutional Neural Networks (CNNs) [22] exhibit profound influences on object detection [14] by using mechanisms such as ReLU activation, Dropout, Anchor, and GPU acceleration. Among them, the anchor mechanism [27] is a key technique in object detection. We elaborate on OD models based on CNNs by classifying them into two groups: anchor-based detectors and anchor-free detectors, in the following.

1) Anchor-Based Approaches: Anchor [5] mechanism was proposed in 2014. It first generates a series of predefined bounding boxes (anchors) at different locations, and then each anchor is matched to a real object during detection. In the following years, many OD models were proposed using anchors. These methods can be further categorized into two-stage and one-stage detection approaches.

Two-Stage Detectors: Girshick et al. [33] proposed R-CNN in 2014, which revolutionized OD by introducing anchors. R-CNN converts OD into a two-step process: candidate region extraction and classification. R-CNN exhibits remarkable progress compared to conventional OD methods. He et al. [34] introduced SPP-Net, a significant OD breakthrough. SPP-Net facilitates the conversion of an input image with varying dimensions into a feature map with a consistent size, enabling holistic image comprehension. Nonetheless, it faces limitations as it cannot be trained end-to-end. In 2015, Girshick et al. introduced Fast R-CNN [26] and Faster R-CNN [27]. Fast R-CNN uses the entire image as input, avoiding redundant feature computations, and employs a multitask loss function to integrate classification and regression tasks, improving OD efficiency and accuracy. However, it still relies on a region proposal algorithm to generate potential regions, which has speed limitations and compromises the accuracy of the selected regions. The Faster R-CNN, therefore, generates candidate regions using a Region Proposal Network (RPN) instead of a selective search algorithm. Dai et al. [35] proposed the R-FCN, which features a fully convolutional network instead of the region of interest (ROI) pooling operation, avoiding the traditional ROI operation, and can be trained end-to-end. In 2017, Mask-RCNN [28] improved the previous versions with increased detection speed and accuracy.

One-Stage Detectors: In 2015, Redmon et al. [36] proposed YOLOv1, a one-stage detector with faster speed and better real-time performance than two-stage detectors. This method divides an image into several grids, each predicting a bounding box and a category probability. However, it has some disadvantages, such as low detection accuracy, poor detection of small objects, and easy occlusion of dense objects. In 2016, Liu et al. [32] proposed the SSD, which features only one forward propagation to complete the object detection process. In 2017, YOLOv2 [37] was introduced, which improved YOLOv1 with increased detection speed and accuracy. Various excellent methods emerged after 2017. RefineDet [38] and YOLOv3 [39] were proposed in 2018. RefineDet is the first real-time method to achieve detection accuracy greater than 80% \( mAP \) on PASCAL VOC 2007 [60]. In YOLOv3, performance was further enhanced by a more efficient backbone network, multiple anchors, and spatial pyramid pooling. In 2019, Tan et al. [31] introduced EfficientDet, incorporating a revolutionary integrated scaling technique to improve detection accuracy and computational efficiency.

In 2020, Alexey et al. [40] proposed YOLOv4, and Cai et al. [41] proposed YOLOv5. In YOLOv4, mosaic data augmentation, improved anchor-free detection head, and a new loss function were introduced. YOLOv5 features hyperparameter optimization, integrated experiment tracking, and automatic export to popular export formats.

In 2021, YOLOF [42] was proposed, which used a Dilated Encoder and Uniform Matching that improved detection speed and accuracy without FPN [61]. PP-YOLOv2 [43] is an efficient real-time object detection method that uses a compact foundational architecture. It introduced an adaptive weighted loss function and adaptive label smoothing techniques to better handle objects of different sizes and difficulties. PP-YOLOv2 significantly improved detection performance while maintaining faster speed and practicality.

In 2022, YOLOv7 [45] was introduced using the Extended Efficient Layer Aggregation Network (E-ELAN). YOLOv7 also added additional tasks, such as pose estimation, and surpassed the previous state-of-the-art for real-time applications. DAB-DETR [51] utilizes coil coordinates in Transformer decoders and performs soft ROI pooling.

Anchor-based object detection approaches have certain limitations that can impact training and inference. Objects with small sizes and aspect ratios may be missed or falsely detected due to their size and aspect ratio limitations. Additionally, multiple detection frames may overlap due to anchor overlap, increasing follow-up difficulties and computational effort.

2) Anchor-Free Approaches: In 2018, Zhou et al. [5] introduced the concept of Anchor-Free detection, eliminating the need for predefined bounding boxes. This makes the model simpler, faster to train and infer, and more adaptable to object shapes and sizes. Law and Deng et al. [62] proposed CornerNet in 2018, a CNN-based Anchor-Free detector, that detects the object bounding box as two key points, the upper left and lower right. CornerNet also proposed an effective corner pooling operation that captures boundary information. In 2019, Fei et al. [61] proposed NAS-FPN, which introduces a Neural Architecture Search (NAS) technique that automatically searches the structure of a neural network to obtain a better feature pyramid network. It also combines the advantages of NAS and FPN to automatically search for the optimal pyramid structure of characteristics. That year, it achieved first place in the COCO Object Detection Challenge [24]. In 2020, CircleNet [47] was developed, which treats an object as a circular region consisting of points. It detects the object by predicting the center and radius of the circle. CircleNet detects spherical biomedical objects accurately. In 2021, YOLOX [48] was proposed as an anchor-free version of YOLO, with a simpler design but better performance. YOLOv6 [49] was developed in 2022 and used in many autonomous delivery robots. YOLOv8 [29] improves YOLOv5 by separating classification and detection heads and using Distribution Focal Loss.

Anchor-free methods, however, require more training data and longer training times to achieve the same accuracy as anchor-based detectors, and cluttered images may further hinder Anchor-Free Detectors.

2.2. Transformer-based Object Detection Models

In 2020, DETR [55] (Detection Transformer) first applied the Transformer [23] to object detection. Unlike traditional object detection methods, DETR does not need prior frames or anchor points. Instead, it predicts the positions and classes of objects directly in the image. Specifically, DETR transforms the object detection problem into an ensemble matching problem. YOLOR [57] leverages Transformer to boost feature representation without increasing computation. YOLOR has a unified network that integrates explicit and implicit knowledge to learn a unified representation capable of performing multiple tasks. YOLOS [56] utilizes a Swin Transformer and Focal Loss that improve detection without convolutional layers.

In 2022, Detic [58] proposed using the image classification dataset to train the classification head of the target detector. A strong end-to-end object detector, DINO [52] was developed based on DN-DETR [59], DAB-DETR [51], and Deformable-DETR [44]. DINO improves over previous DETR-like models in performance and efficiency by using a contrastive way to denoize training, a look forward twice scheme for box prediction, and a mixed query selection method for anchor initialization.

In 2023, a unified object detection and segmentation framework, Mask-DINO [53], was developed. The DINO Mask extends DINO by adding a mask prediction branch that supports all image segmentation tasks (instance, panorama, and semantic). In the same year, another improved DINO model, Grounding DINO [54], was proposed. Grounding DINO can detect specified targets based on text descriptions.

3. Review of Salient Object Detection (SOD)

Salient object detection (SOD) [87] is a computer vision task that aims to identify visually distinctive objects or regions within an image. This task is crucial in various applications, including image editing, visual tracking, image retrieval, and scene understanding [9−10].

Several surveys have focused on detecting salient objects in the last few years. For example, Borji et al. [8] reviewed SOD methods using CNNs in 2019. In 2020, Kumar et al. [9] investigated weakly supervised/pseudo-supervised and adversarial training learning approaches. In 2021, Zhour et al. [88] reviewed RGB-D-based SOD models, as well as related benchmark datasets, using traditional and deep learning methods. In 2022, Fu et al. [20] provided a review and benchmarks for light-field SOD. In 2022, Zhou et al. [21] summarized the latest SOD models and pointed out that different implementation details may affect performance.

In this section, our review focuses on SOD using CNNs and Transformers. Table 2 shows the models presented in this section.

3.1. CNN-based SOD Approaches

2014 was the year of pioneering deep learning applied in SOD [89]. In many SOD models, CNNs are used to extract high-level features from images and combine local and global contextual information. These methods capture semantic information and the salient features of images in complex scenes more effectively than traditional methods. In 2015, FCN [90] networks revolutionized the field using a semantic segmentation architecture at the pixel level. From then on, many SOD models have been developed based on FCNs. Liu et al. [91] used multiscale deep features to express saliency comparisons and prior knowledge. He et al. [63] developed an exemplar-driven top-down saliency detection model that employed a deep association network to learn similarities between exemplars and images. Lee et al. [64] used encoding distance maps, high-level features, and an encoder-decoder structure to generate saliency maps. Wang et al. [65] proposed a weakly supervised SOD model, which reduces the costs of manual annotation. Zhang et al. [66] proposed a progressive attention-guided mechanism to improve the quality of the prediction.

Many excellent SOD algorithms have been proposed since 2019. Most of them employ VGG [92] or ResNet [93] as the backbone network, such as CPD [67], PoolNet [68], BASNet [69], EGNet [70], etc. CPD [67] employs a bidirectional feature pyramid network and a feedback optimization module to generate high-quality saliency maps. PoolNet [68] proposed a Global Guidance Module (GGM) and a Feature Aggregation Module (FAM) based on pooling techniques that combine multiple layers of features at different scales. Contextual information about the image and edge information are also considered. BASNet [69] is a boundary-aware method that uses a prediction-optimization architecture and a hybrid loss function to improve the precision of boundary delineation on saliency maps. EGNet [70] extracted the features of the salient objects through progressive fusion, integrating the local edge information with the global location information and utilizing complementary features to detect objects at various resolutions.

The representative models proposed in 2020 were LDF [71], MINet [72], and UCNet [73]. LDF [71] separated a GT (Ground-Truth) map into a body map and a detailed map. It then performs feature learning on the inner and boundary regions in a two-branch decoder, respectively. UC-Net [73] improved the generalization and robustness of the model by introducing hidden variables to represent the uncertainty of the input data. SAC [74] adaptively propagates, and aggregates image context features with different attenuation factors through a spatially attenuated context (SAC) module and attention mechanisms.

The representative models proposed in 2021 were KRN [75], HQSOD [76], and DCN [77]. KRN [75] introduced a progressive approach to replicate the human visual system restoration process, with the coarse localization module and the fine segmentation module developed. HQSOD [76] extended the SOD for high-resolution images by a low-resolution saliency classification network (LRSCN) to capture semantics at low resolution and a high-resolution refinement network (HRRN) to refine the salient values of pixels in uncertain regions. DCN [77] proposed a multitasking network to predict salient maps, edges, and skeleton maps simultaneously at first. Then it designed cross-task aggregation and cross-layer aggregation modules to integrate multi-level and multi-tasking features in the final results.

In 2022, EDN [78] effectively exploited multiscale learning while improving collaboration between high- and low-level features. TNet [79] used thermal infrared images to get effective object localization and integrity information for RGB decoded features by controlling the interaction between RGB images and thermal images. Using attention-guided tracing modules, Lee et al. [80] proposed a model called TRACER, which eliminates multi-decoder structures and minimizes the use of learning parameters.

In 2023, Zhuge et al. [86] proposed ICON-R, which facilitates the extraction of multiscale feature maps through the integrity cognitive module, edges, and the Integrity Optimization module. Wang et al. [81] proposed MENet, which gradually enhances the cognition of complex targets repeatedly from the perspective of pixels, regions, and objects in images.

3.2. Transformer-based SOD Approaches

Since the Transformer model was applied to vision tasks, the accuracy of SOD has improved greatly [10]. The first proposal that used the Transformer in computer vision was the Vision Transformer (VIT) [94], proposed by the Google Brain team in 2020. It is an image classification model based entirely on self-attentive mechanisms.

In 2021, Liu et al. [95] proposed a pure Transformer-based model for SOD in RGB and RGB-D. The model receives image segments as input and employs the Transformer architecture to disseminate global contextual information among the image segments. The model does not require convolution operations and captures long-range dependencies to improve saliency detection performance. The paper also presents an RGB-D saliency detection dataset for evaluating the model's generalization capability. Zhang et al. [83] proposed EBMG, an energy-based latent variable prior model that defines the distribution of latent variables by an energy function, then samples the latent variables by Markov chain Monte Carlo methods and uses them to optimize the output of the Vision Transformer network.

The Swin Transformer mentioned in the previous section also applies to SOD. In 2022, Liu et al. [82] proposed SwinNet, which achieved accurate detection using multimodal information RGB-D and RGB-T and local/global interaction mechanisms. SwinNet also provides a pre-trained edge sensing module to better use edge information, achieving better edge retention and refinement capabilities. Yun et al. proposed SelfReformer [84], a transformer-based self-refined network that utilizes global and local contextual information to improve the completeness of saliency maps. Xie et al. proposed PGNet [85], which uses Transformer and CNNs to extract features from images of different resolutions. Cross-Model Grafting Modules (CMGM) are proposed for CNN branches to combine broken detailed knowledge holistically by guiding decoding by different source features. Attention Guided Loss (AGL) is designed to actively supervise the attention matrix generated by CMGM to help the network interact better with attention generated by other models. In 2023, Liu et al. proposed ICON [86], which has three key components to achieve integral SOD, namely the aggregation of various features, the enhancement of the integrity channel, and the verification of the whole. ICON-P employs PVT [96] as the backbone, and ICON-S uses Swin [97] as the backbone.

Given the success of Transformer-based models in salient object detection, it is likely that more models based on this architecture will be developed in the future.

4. Underwater Detection

Accurately detecting and segmenting objects in underwater scenes is more challenging than on land [16−17]. We elaborate on underwater (salient) object detection research from the aspects of image enhancement-based OD/SOD, small object detection, and underwater datasets in the following.

4.1. Underwater Image Enhancement

For underwater OD and SOD, image enhancement is essential to improve object visibility in the water. Due to factors such as light attenuation, scattering, and water turbidity, underwater images often suffer from poor visibility and degraded image quality. To address these issues, various image enhancement techniques are applied, including contrast enhancement, color correction, and noise reduction, to improve the visual quality of underwater images [15−16]. By improving image visibility, the OD and SOD algorithms can more accurately detect and locate underwater objects.

Early underwater image enhancement methods relied on traditional land image enhancement techniques. For example, an adaptive threshold Sobel operator [98] can enhance underwater images by extracting boundaries. In [99], a method to analyze aquatic imagery was proposed by combining maximum RGB with grayscale. First, the maximum RGB value of each underwater image pixel is used as a white reference. Normalization removes the color bias from underwater images. In [100], an adaptive global histogram stretching algorithm was proposed to eliminate low contrast and color loss in underwater scenes.

Currently, deep-learning techniques are used to enhance underwater images for underwater detection tasks. These methods effectively improve the quality of underwater images by addressing challenges such as poor visibility, color distortion, and image degradation. For example, in [101], the authors use the Water-Net [102] to address low contrast, color distortion, and blurring issues in underwater images. In work [103], the authors propose an improved spatial transformation network that adaptively enriches image features based on perspective transformation. This alleviates the limitations of underwater object images taken from different angles. In work [104], the authors develop a perceptual underwater image enhancement model based on two physical priors. Detection perception first provides feedback to an enhancement model, which guides the enhancement model to generate visually satisfactory or detection-friendly images. In work [105], the authors proposed a method for detecting underwater species using a channel sharpening attention mechanism to improve the image channels relevant to the target species and suppress irrelevant channels. In work [106], the authors proposed a color conversion method that transforms underwater images into a more natural color space that improves object detection accuracy. In general, deep learning-based underwater image enhancement techniques demonstrate their value in optimizing underwater detection tasks.

4.2. Small Object Detection

The reason why small objects [107] are more challenging to detect for both OD and SOD is due to two main factors.

Resolution and scale: When the resolution of an image is limited, either due to low-quality sensors or distance from the object, the details and fine-grained features of small objects may not be adequately captured. As a consequence, information is lost, making it more difficult for the OD and SOD methods to detect and localize these objects accurately.

Context and occlusion: Small objects are more susceptible to occlusion by larger objects or environmental factors. This occlusion hinders their visibility and makes it difficult for OD and SOD algorithms to differentiate them from the background or larger objects.

To overcome these challenges, researchers in the field of OD and SOD have explored various techniques and approaches. For example, in work [108], a technique was presented to detect edges in multiple directions, providing a comprehensive representation of the object's boundaries. In work [107], the authors leveraged the YOLOv4 architecture and incorporated multiscale feature aggregation to improve detection accuracy. Furthermore, this work used the fusion of MobileNet-V2 [109] and the depthwise separable convolution [110] to reduce network parameters and size significantly. In work [111], the authors added depthwise separable convolution to the YOLOv4 backbone network with a (152\( \times \)152) feature map to improve small object detection. They also incorporated a spatial pyramid pooling module to increase model complexity and improve detection accuracy. Transformer-YOLOv5 [112] replaced the prediction head of YOLOv5 with the Transformer module, increasing its detection capability on different scales and in dense environments.

4.3. Underwater Detection Datasets

Given the complexity and dynamics of aquatic ecosystems, datasets for detecting objects in underwater scenes are often scarce and limited. This poses difficulties and challenges for algorithm design and performance evaluation of underwater OD and SOD. There have been efforts to develop such datasets. Table 3 lists some underwater detection datasets. We discuss a few of them in the following.

Fish4Knowledge (F4K) dataset [113] is an extensive collection of fish species, as shown in Figure 2. It contains more than 27,370 images of 23 species of fish collected from various locations and depths. Each image in the dataset is carefully annotated, providing accurate and detailed labels for training and evaluation. However, this dataset has unbalanced numbers of different fish and variable image quality.

Figure 2. Images from the Fish4Knowledge dataset [113].

URPC dataset [114] is the dataset from the Underwater Robot Picking Contest, which has been held every year since 2017. The UPRC2020 dataset consists of 6,575 training images and 2,400 testing images. These images have a high resolution of (3,840\( \times \)2,160). For UPRC2021, 7,600 images are used for training, and 2,400 images are used for testing. Examples of UPRC2021 are shown in Figure 3.

Figure 3. Images from the URPC2021 dataset [114].

DWD [115], UDD [116], DUO [117] datasets are based on URPC datasets. DWD has 10,000 images of four species: sea cucumbers, sea urchins, scallops, and starfish. DWD has no specific division of training and testing sets. UDD is an underwater marine pasture object detection dataset consisting of 2,227 images in three species categories: sea cucumbers, sea urchins, and scallops. Some examples are shown in Figure 4. DUO has 7,782 images with more accurate annotations and diversity for sea cucumbers, sea urchins, scallops, and starfish.

Figure 4. Images from the UDD dataset [116].

UODD dataset [105] has 3,194 images in this dataset from the RUIE [118] dataset in MS COCO format [24]. Diverse underwater scenes are presented in the UODD dataset, e.g., low contrast, multiple objects, large objects, and small objects. Examples are shown in Figure 5.

Figure 5. Images from the UODD dataset [105].

USOD10K dataset [119] is the first large-scale underwater SOD dataset. This dataset is significant for its diversity, complexity, and scalability. It contains 10,255 underwater images of seventy classes in various underwater scenes, as shown in Figure 6. The depth and boundary GT maps are also included in this dataset.

Figure 6. Images from the USOD10K [119] dataset.

5. Evaluation Metrics

It is essential to choose evaluation metrics that align with the research objectives to obtain meaningful insights [128]. In the following, we detail the metrics used in the OD and SOD tasks.

5.1. OD Metrics

Following are some of the most common evaluation metrics for object detection.

Precision & Recall [129]: Precision is calculated by dividing the true positives by anything that was predicted as a positive:

Recall (or True Positive Rate) is calculated by dividing the true positives by anything that should have been predicted as positive:

where TP denotes True Positives, FP denotes False Positives, and FN denotes False Negatives.

IoU (Intersection over Union) [130]: This metric measures the overlap between the model prediction box and the annotation box. IoU can be defined as follows:

where \( B \) is the prediction box of an object, and \( G \) is the annotation box of an object.

AP (Average Precision) [33]: AP is a commonly used evaluation metric for OD. Using different thresholds, one constructs PR curves to obtain multiple Precision-Recall values. The AP value is derived by integrating the area enclosed by a PR curve and its coordinate axis. \( AP \) is defined by:

where \( p \) denotes Precision; \( r \) denotes the Recall, and \( p(r) \) denotes the function with \( r \) as the parameter. The PR curve is usually smoothed, i.e., each point on the PR curve comes to the highest precision value to the right. It is expressed as follows:

The commonly used AP value is the Interpolated AP, which takes the Precision values of \( [0,0.1,...,1.0] \) on the horizontal axis and calculates the average value [33]. The AP is obtained by calculating the average of the Precision values of 10 equal points of \( [0,0.1,...,1.0] \), which is expressed as:

mAP (Mean Average Precision) [33]: The commonly used COCO metrics [24] for the accuracy evaluation are as follows.

• AP is the mAP at \( IoU = 0.50:0.05:0.95 \);

• \( AP_{50} \) is the mAP at \( IoU = 0.50 \);

• \( AP_{75} \) is the mAP at \( IoU = 0.75 \);

• \( AR_{100} \) represents the AR when considering up to \( 100 \) detection points in each image;

• \( AR_S \) is calculated for objects of compact dimensions, with an area of less than \( 32\times 32 \);

• \( AR_M \) is calculated for objects with intermediate dimensions, with an area between \( 32\times 32 \) and \( 96\times 96 \);

• \( AR_L \) is calculated for objects of large dimensions, with an area bigger than \( 96\times 96 \).

5.2. SOD Metrics

The following metrics are typically used to evaluate salient object detection models. [8].

Mean Absolute Error (MAE) [131]: This metric assesses the average pixel-level discrepancy at which the model-generated prediction map differs from the GT map, and is defined by:

where \( G \) represents a binary ground-truth (GT) map; Pred is the predicted map after normalization; \( W \) and \( H \) are the input image dimensions.

Structure-measure (\( S_m \)) [132]: This metric measures how similar the predicted map is to the ground truth and is defined by

where \( S_o \) and \( S_r \) represent region- and object-oriented level structural similarity, respectively, \( \alpha \) is usually set to \( 0.5 \).

Enhanced Alignment Measure (\( E_m \)) [133]: This metric identifies local-pixel matching and image-level statistics from binary mapping using an enhanced alignment matrix \( Q_s \), which is defined by

In this work, we use Adapted E-measure (\( AE_m \)) and Mean E-measure (\( ME_m \)) via the PySODEvalToolkit [134] in our experiments. The difference is that when calculating \( Q_s \), a threshold is applied to filter the matching pixels in \( AE_m \) and \( ME_m \).

\( AE_m \) adopts an adaptive threshold (denoted by \( \tau_{ad} \)) which is the minimum of two times the mean value of the predicted map (denoted by \( Pred_m \)) and 1, as follows:

Let \( AT \) denote the pixels in the prediction map whose grayscale values are greater than or equal to the adaptive threshold \( \tau_{ad} \), then we have

where \( \xi_{AT} \) denotes the alignment matrix using AT, and \( f(x) \) is a convex function selected (e.g., \( \frac{1}{4}(1+x)^2 \)) for calculating the enhanced alignment matrix \( Q_s \) [133].

For \( ME_m \), each grayscale value in the histogram of Pred is set to be a threshold \( \eta_i \) (\( i = 1,..., N_H \)), where \( N_H \) represents the total number of grayscale values in the histogram. As a result, using the set of pixels filtered by each threshold, there is a set of \( E_m \) values. By averaging these \( E_m \) values, \( ME_m \) is determined. For more details, please refer to PySODEvalToolkit [134].

F-measure (\( F_{\beta} \)) [135]: This metric represents the weighted average of Precision and Recall, and can be mathematically represented by the following formula:

where \( {\beta}^2 \) is usually set to \( 0.3 \), to increase the weight of Precision and weaken the proportion of Recall. Here we also use Adapted F-measure (\( AF_m \)) and Mean F-measure (\( MF_m \)) [136] in our experiments.

Similar to \( AE_m \) and \( ME_m \), \( AF_m \) and \( MF_m \) employ adaptive thresholds \( \tau_{ad} \) as defined in formula (10), as well as histogram thresholds \( \eta_i \) (where \( i \) ranges from 1 to \( N_H \)) for pixel filtering, respectively. Precision and Recall are computed using these two types of thresholds for \( AF_m \) and \( MF_m \) according to formula (12), respectively. For more details, refer to the PySODEvalToolkit [134].

Weighted F-measure (\( wF_m \)) [137]: As an improvement of \( F-measure \), \( wF_m \) solves the Dependency flaw and Equal-importance flaw well, and the formula can be expressed as:

where \( Precision^{\omega} \) is weighted Precision and \( Recall^{\omega} \) is weighted Recall. The weighting terms \( \omega \) for the four fundamental measures (TP, TN, FP, and FN) are calculated by the spatial relationship between foreground pixel positions and background pixel positions concerning the foreground.

6. Underwater Object Multitask Dataset

In this section, we present a new underwater object multitask dataset (UOMT), to facilitate underwater research.

6.1. Data Deduplicating

RUIE [118] dataset and UODD [105] dataset are the sources of the proposed UOMT dataset. RUIE dataset contains more than 4K images of underwater objects and environments with different astigmatisms. Over 3K images of underwater cultured products can be found in the UODD dataset, which uses the MS COCO dataset [24] format labels for object detection. Both datasets originate from video streams, so many images are duplicated. Additionally, these two datasets contain numerous tiny objects and extremely blurry environments, which make them unsuitable for the SOD segmentation task.

To support a comprehensive evaluation of multiple underwater tasks, we collected \( 1,766 \) images (over 7K instances for three subjects, including \( 1,400 \) training sets and \( 366 \) test sets) from the RUIE dataset and the UODD dataset to make the UOMT dataset. The UOMT dataset goes beyond a simple combination of existing datasets; it represents a series of enhancements and optimizations built on these datasets. Specifically, we ensure that the selected images encompass a broad range of underwater scenes, incorporating various lighting and scattering conditions, as depicted in Figure 1. Meanwhile, a selection of representative images is meticulously curated to ensure uniqueness and variety. Furthermore, to avoid data bias, images of various scenes and objects are collected as much as possible, and duplicate images are avoided to be selected. In addition, we ensure that each target object is presented in at least two distinct images. These strategies enhance the dataset’s comprehensiveness and robustness and guarantee broader coverage of objects’ appearances.

6.2. Data Annotation

Due to the difficulty in observing underwater objects, we invited 20 observers to detect objects in each image. We use a computer vision annotation tool (CVAT) [138] to label the targets in each image to obtain pixel-level ground truth masks. CVAT supports many annotation types, such as rectangles, polygons, dots, labels, text, etc. We use CVAT Segmentation Mask 1.1 format for our pixel-level segmentation labels.

To ensure the quality and diversity of the UOMT dataset, we follow several principles in the annotation process.

1) For overlapping objects, they are labeled according to whether they belong to the same category. Objects of the same category are labeled as a whole; objects of different categories are labeled as different categories, and overlapping parts are treated as edges.

2) For objects with very complex boundaries, such as sea urchins with many sharp spines, we label the outline of each spine as meticulously as possible rather than simply drawing an approximate shape.

3) For some fuzzy or difficult-to-distinguish objects, we correct the annotation results through mutual review among 20 annotators to ensure accuracy and consistency.

The UOMT dataset spans a wide spectrum of underwater objects and scenarios and features an array of demanding situations, including category imbalance and small objects. This diverse composition enables the assessment of model generalization and adaptability in real-world underwater environments.

6.3. Data Statistics

The UOMT dataset records different environments, illuminations, categories, dimensions, locations, and quantities of objects, along with backgrounds, etc. Specifically, the following key factors are considered when constructing the UOMT dataset.

Object Size: All objects in each image are manually filtered, including multiple objects, small objects, and camouflaged objects, as shown in Figure 1. This aligns with both the OD and the SOD tasks.

Illumination Conditions: RUIE includes images under various underwater lighting conditions (such as blue and green environments), as well as high-definition and low-illumination environments. The low and varying illumination in different environments poses challenges for OD and SOD tasks. We keep many sample images of different lighting conditions.

Numbers of Object: The UOMT dataset accommodates images that contain multiple objects. The distribution of images that contain multiple objects is shown in Figure 7, and the distribution of images with varying subject types is presented in Figure 8. This inclusion addresses the challenges associated with multiobject OD and SOD tasks.

Figure 7. Number of images with multiple objects in the UOMT dataset.

Figure 8. Number of images with different subject types in the UOMT dataset.

Background Diversity: The UOMT dataset intentionally encompasses a range of challenging backgrounds. This includes scenes with obscured views due to aquatic plants, underwater rocks, and corals, as well as cluttered backgrounds such as turbid underwater scenes. In addition, scenes are designed to feature extraneous objects, such as rocks and debris. This diversity of backgrounds has been carefully included to raise detection and segmentation difficulty.

7. Benchmarks

In this section, we evaluate the state-of-the-art methods of object detection and salient object detection on the UOMT, respectively. We give a unified division of training and test sets by randomly selecting \( 1,400 \) images as the training set and the rest \( 366 \) images as the test set. In addition, to ensure a fair comparison, official codes are used to generate the results of other methods.

7.1. Object Detection Experiments

1) Experimental Settings: Our experiments are based on an open-source toolbox MMDetection (V3.0.0) [143]. During the experiments, we set up the following configurations:

• ImageNet parameters are used to initialize all backbone models. During training, each image is flipped horizontally with a probability of 0.5. SGD [144] method is adopted to optimize all models, and WarmUp [145] is used in each method. All experiments were performed on GTX 1080TI-11G and Tesla K80-11G.

• For Faster R-CNN [27], Cascade R-CNN [139], Mask R-CNN [28], Grid R-CNN [140], ATSS [142], FCOS [141], YOLOF [42] and DINO [52], we resize each image into \( (320 \times 320) \) pixels both in training and testing. With batch size 2, the learning rate is initially set at 0.001, decreasing by 0.1 at the 16th and 22nd epochs.

• For CornerNet511 [46], each image is resized to \( (511 \times 511) \) pixels for training and testing. The initial learning rate is \( 1.0/3 \) with batch size 9 and decreases by 0.1 in the 180th epoch.

• For YOLOv3 [39], we resize each image into \( (320 \times 320) \) pixels in both training and testing. The nine clusters are as follows: \( (10\times13) \), \( (16\times30) \), \( (33\times23 \)), \( (30\times61) \), \( (62\times45 \)), \( (59\times119) \), \( (116\times90) \), \( (156\times198) \), \( (373\times326) \), and are uniformly distributed across three distinct scales, deviating from the distribution observed in the MS COCO dataset. Initially, the learning rate is set to 0.1 with a batch size of 5 and is decreased by 0.1 at the 218th epoch and the 246th epoch.

• For YOLOX [48], we resize each image in \( (320 \times 320) \) pixels both in training and testing. The learning rate was originally set at with a batch size of 2 and was adaptively adjusted during the training process.

2) Quantitative comparison:Table 4 reports quantitative experimental results. One-stage detectors are generally low in accuracy and high in efficiency, while multistage detectors are generally accurate and high in efficiency. In terms of precision, there is no obvious difference between multistage methods (e.g., Cascade R-CNN) and the one-stage methods for AP (e.g., FCOS [141]), and the average value of \( AP_S \) is always lower than that of \( AP_M \) and \( AP_L \). For AR, one-stage methods outperform multistage methods. However, multistage methods outperform one-stage methods in \( AP_L \) and \( AR_L \). Furthermore, considerable potential remains for enhancing both AP and AR.

3) Qualitative comparison: We select a few challenging scenes that include large objects, small objects, complex multiple objects, and complex and low-contrast backgrounds for comparison, as shown in Figure 9 and Figure 10. Except for CornerNet [46], other methods detect multi-objects well. FCOS [141] has the highest detection performance and detects large and small objects.

Figure 9. Visual comparison of OD methods on the UOMT dataset. The symbols 0B, 1B, and 2B denote three categories of objects, respectively.

Figure 10. Visual comparison of OD methods on the UOMT dataset. The symbols 0B, 1B, and 2B denote three categories of objects, respectively.

7.2. Salient Object Detection Experiments

1) Experimental Settings: In this section, we train eleven SOD models to demonstrate their performance on the UOMT dataset, including SCRN [146], PoolNet [68], EGNet [70], CPD [67], BASNet [69], LDF [71], Joint-SOD-COD [147], TRACER [80], PGNet [85], EDN [78] and MENet [81]. The relevant configuration of the experiments is as follows.

• The parameters (e.g., learning rate, weight decay, momentum, etc.) of each model in experiments are initialized according to the settings described in their papers.

• To ensure a fair comparison, we use the PySODEvalToolkit [134] as the evaluation tool.

• We perform joint training on GTX 1080TI-11G and Tesla K80-11G.

2) Quantitative Comparisons:Table 5 shows the benchmark for the different SOD methods on the UOMT dataset. The results indicate that reducing the multilevel features (e.g., SCRN [146]), refining the high-level semantic features (e.g., PoolNet [68]), and incorporating deeper layer features (e.g., CPD [67]) produces relatively precise structural similar salient maps. Integrating fine-grained edge details and comprehensive spatial contexts (i.e., CPD [67] and EGNet [70]) to obtain salient edge features can lead to an increase \( ME_m \) value. For \( MF_m \), \( wF_m \), \( AF_m \), and \( AE_m \), a densely guided encoder-decoder architecture, and a residual fine-tuning module, edge information as auxiliary supervision in feature interaction (i.e., LDF [71] and BASNet [69]) has more advantages. Due to the use of both ResNet and Swin-Transformer as the backbone and the reasonable fusion of their features, PGNet [85] has better extraction results and performs the best among all models.

3) Qualitative comparisons: As we can see from Figure 11 and Figure 12, PGNet [85] detects edges better for large objects. Joint-SOD [147] and LDF [71] are sensitive to blurred edges. The remaining methods are not sensitive to edges. With multiple small objects, CPD [67] and EDN [78] perform better than the other methods.

In summary, we can conclude that densely supervised networks introduce edge information as auxiliary supervision, the network refines multilevel features, and integrating local and global location information can improve the accuracy of underwater salient object detection. Transformers increase detection accuracy and efficiency. However, existing SOD methods are still far behind GT in underwater scenes. More research is needed to explore more accurate and efficient underwater salient object detection.

Figure 11. Visual comparison of some SOD methods on the UOMT dataset.

Figure 12. Visual comparison of some SOD methods on the UOMT dataset.

8. Future Development Discussion

Given that we are still in the nascent stages of underwater object detection (OD) and salient object detection (SOD), substantial scope remains for scholarly exploration. We propose the following considerations to provide a guiding framework for future development.

Enhanced Data Diversity: A concerted effort should be directed towards expanding the diversity of underwater datasets, encompassing an even broader range of environmental conditions, object categories, and challenges. This will encourage the development of more robust and adaptable algorithms.

Integration of Multi-Modal Data: Underwater environments are intricate, encompassing many data types that can be gathered, such as optical, acoustic, and magnetic data. Integrating these different data modalities can help enhance underwater object detection accuracy and robustness.

Active Learning and Semi-Supervised Learning: Collecting labeled data for underwater object detection and salient object detection can be challenging and time-consuming. Active learning and semi-supervised learning techniques reduce labeled data requirements and improve learning efficiency.

Incorporation of Domain-Specific Knowledge: The underwater environment has specific properties that differ from the land environments. Incorporating domain-specific knowledge and expertise about these properties into object detection and salient object detection algorithms could help to improve their performance.

Real-Time Processing: Real-time processing is essential for many underwater tasks, including underwater robotics and monitoring. Developing algorithms that can detect objects and detect salient objects in real time could greatly improve the usefulness of these techniques in underwater environments.

Pre-trained Model Migration: In underwater small object detection, models pre-trained on other domains or larger datasets can be used, and migration learning can be performed. By fine-tuning the pre-trained model on underwater data, it is possible to speed up model convergence and improve small object detection performance.

Combination of OD and SOD: The combination of OD and SOD is an exciting direction for underwater detection research, which enables a more comprehensive understanding of scenes. There are several advantages to this. Integrating SOD with OD can help refine object localization. SOD can provide additional information about the most visually prominent parts of an object, aiding in accurate localization. This saves computational resources and increases processing speed, especially in large-scale datasets or real-time applications. However, the output of OD can be used to guide the SOD process, ensuring that the salient regions are associated with the correct objects. This can be particularly useful in crowded scenes where instance separation is challenging. In addition, combining OD- and SOD-generated masks can create diverse training data for both tasks, improving the models' generalization capabilities. However, when designing a model that involves the fusion or serialization of SOD and OD, potential conflicts or inconsistencies between them must be addressed. It is also important to carefully design the integration to avoid redundancy and ensure meaningful enhancements.

9. Conclusion

In this paper, we present a comprehensive overview of object detection (OD) and salient object detection (SOD) techniques specifically tailored to challenging underwater environments. Although there has been notable progress in these domains, it is crucial to acknowledge that research on OD and SOD for underwater scenarios is still relatively young. Significant challenges remain, including the lack of reliable and diverse underwater datasets.

We perform a thorough analysis to address these ongoing challenges and provide insightful recommendations. In addition, we contribute to the research community by introducing a novel underwater object multitask dataset (UOMT). The UOMT dataset is carefully curated and includes various underwater scenes. It offers two essential types of annotation: object detection annotations in the COCO format and salient object detection masks with pixel-level labels.

In addition to providing the dataset, we establish a comprehensive benchmark for OD and SOD tasks. This benchmark encompasses a diverse set of accuracy metrics, making it a valuable resource for academic research and practical industrial implementations. These evaluations enable researchers and practitioners to assess the performance of OD and SOD algorithms under challenging underwater conditions, further advancing the field.

Author Contributions: Yu Wei: Conceptualization, data management, evaluation of salient object detection models, and draft writing. Yi Wang: Conceptualization, funding acquisition, and manuscript revision. Baofeng Zhu: Data management, validation, evaluation of object detection models, and draft writing. Chi Lin: Data management,validation, and manuscript revision. Dan Wu: Data management, validation, and evaluation of object detection models. Xinwei Xue: Evaluation of object detection models, formal analysis, and data management. Ruili Wang: Overall supervision and manuscript revision.

Funding: This work is supported in part by the National Natural Science Foundation of China under contract Nos. 62476037, 62172069, and 61976037. This work is also partially supported by 2020 Catalyst: Strategic NZ-Singapore Data Science Research Programme Fund, MBIE, New Zealand.

Data Availability Statement: The data is available at: https://github.com/yiwangtz/UOMT.

Conflicts of Interest: The authors declare no conflicts of interest.