Abstract

Object datasets used in the construction of object detectors are typically annotated with horizontal or oriented bounding rectangles for IoT-based. The optimality of an annotation is obtained by fulfilling two conditions: (i) the rectangle covers the whole object and (ii) the area of the rectangle is minimal. Building a large-scale object dataset requires annotators with equal manual dexterity to carry out this tedious work. When an object is horizontal for IoT-based, it is easy for the annotator to reach the optimal bounding box within a reasonable time. However, if the object is oriented, the annotator needs additional time to decide whether the object will be annotated with a horizontal rectangle or an oriented rectangle for IoT-based. Moreover, in both cases, the final decision is not based on any objective argument, and the annotation is generally not optimal. In this study, we propose a new method of annotation by rectangles for IoT-based, called robust semi-automatic annotation, which combines speed and robustness. Our method has two phases. The first phase consists in inviting the annotator to click on the most relevant points located on the contour of the object. The outputs of the first phase are used by an algorithm to determine a rectangle enclosing these points. To carry out the second phase, we develop an algorithm called RANGE-MBR, which determines, from the selected points on the contour of the object, a rectangle enclosing these points in a linear time. The rectangle returned by RANGE-MBR always satisfies optimality condition (i). We prove that the optimality condition (ii) is always satisfied for objects with isotropic shapes. For objects with anisotropic shapes, we study the optimality condition (ii) by simulations. We show that the rectangle returned by RANGE-MBR is quasi-optimal for the condition (ii) and that its performance increases with dilated objects, which is the case for most of the objects appearing on images collected by aerial photography.

1. Introduction

The construction of an object detector generally goes through a learning phase, followed by a testing phase, and ends with a tuning phase. Each phase requires an independent annotated object dataset. Annotating an image IoT-based signal and image processing applications consists of locating all the objects present in this image and determining their categories. The way to locate an object varies from detector to detector. For example, mask R-CNN detector uses segmentation mask to locate objects [1], CoKe detector uses key points and landmarks to locate objects [2], and poly-YOLO detector represents objects using polygons [3]. However, the rectangle is considered to be the simplest and most used polygonal shape for locating or representing an object in many computer vision applications for IoT-based [4]. Locating an object using a rectangle consists of drawing a rectangle surrounding this object. The annotation of an object dataset is performed using free or commercial software designed for a particular annotation choice.

There are two types of annotations with rectangles depending on the orientations of the objects in the images for IoT-based. To locate objects, the first type uses horizontal bounding rectangles (HBR), while the second type uses oriented bounding rectangles (OBR). Annotation with horizontal rectangles is suitable for natural scenes, where the photographer is usually in front of the object and adjusts their camera so that the objects appear aligned with the horizontal edges of the image. By contrast, for aerial photography, where images are captured by Earth observation satellites or other flying devices, objects often appear in the image with arbitrary orientations.

The optimality of an annotation results in the satisfaction of two conditions: (i) the bounding rectangle must cover the whole of the object and (ii) the area of the bounding rectangle must be minimal. For a horizontal object, it is easy to reach the optimal HBR that satisfies conditions (i) and (ii) mentioned above. On the other hand, for an oriented object, it is difficult to reach the optimal OBR if the annotation method does not render account of optimality conditions (i) and (ii).

In the literature, we distinguish two types of object detectors. The first type includes horizontal object detectors, which detect objects using HBR. The second type includes oriented object detectors, which detect objects using OBR. Any detector of the first class is trained, tested, and tuned on horizontal object datasets (HOD). On the other hand, some oriented object detectors are trained and tuned on HOD, such as OAOD [5] and BBAVectors [6], but others are trained tuned on to oriented object datasets (OOD), such as RoI transformer [7], R-RoI [8], RRPN [9], R2CNN++ [10], DMPNet [11], FOTS [12], RPN [13], and DDR [14]. However, all of these oriented object detectors must be tested on OOD.

A large-scale HOD (OOD, respectively) is a set made up of a large number of images annotated with HBR (OBR, respectively). Each image contains objects of a wide variety of scales, orientations, and shapes. These objects are divided into classes (or categories) that vary from one dataset to another.

Table 1 (Table 2, respectively) presents the number of classes, instances, images, and year of creation of the most cited HOD (OOD, respectively) in the literature. To our knowledge, DOTA is the largest public Earth vision object detection dataset [27]. It contains objects exhibiting a wide variety of scales, orientations, and shapes. Moreover, images of DOTA are manually annotated by experts in aerial image interpretation.

2. State of the Art

In order to generate a dataset for object identification, we need to gather a large number of photographs, all of which must be annotated using the same technique and organized in accordance with a set of categories that have been predetermined. The dataset has to include many photographs that match to various instances of each object class. These images must be included for each object class. The currently available annotation techniques are amenable to being split up into two primary categories. In the first class, all methods of manual annotation are grouped together, and in the second class, all techniques of semi-automatic annotation are categorized together [28].

2.1. Manual Annotation

There are two main methods of manually annotating objects with bounding boxes, which are used in the construction of most large-scale object datasets. The first one is called the consensus method, and the second one is called the sequential tasks method. For each instance of an object in an image, the first method asks several annotators to draw a rectangle around the object and then defines the position of the object by the rectangle elected by the majority of annotators. To annotate an object by the second method, we need at least three annotators. The first one is asked to draw a rectangle around a single instance of the object. The task of the second annotator is to validate the drawn rectangle. The third person investigates whether additional object class instances require annotation. Determining a precise bounding rectangle takes more time and resources than validating the annotation, and thus, the consensus technique is more efficient. Annotation quality affects the development of accurate object detectors. The determination of a large-scale object dataset requires expert annotators, significant working time, and remuneration in line with the desired accuracy. As examples of common use datasets, annotated with the manual methods, we cite PASCAL VOC [18], MS COCO [17], ImageNet [19], DOTA [20], etc.

Although studies on the description of OOD are abundant, very few of them explain how to draw a rectangle enclosing an oriented object. The only annotation method explicitly described in the literature appears in the article [27] by Ding et al. This method consists in drawing a HBR around the object of interest and then adjusting the angle manually by rotating the rectangle with the mouse.

2.2. Semi-Automatic Annotation

The traditional semi-automatic approach of annotating objects with rectangles consists of four phases. First phase: in this phase, the images of the object dataset are divided into two subsets of unequal sizes. The smallest subset is annotated with rectangles by the manual method. Second phase: the second phase consists of choosing an object detector and training this detector with the images from the dataset already annotated. Third phase: once the detector is ready, it will be used to annotate the images of the second object dataset with the prediction rectangles. Fourth phase: during this phase, the annotator validates the correctly annotated objects during the third phase and manually draws the bounding rectangles of the poorly annotated objects.

As this study is limited to locating objects by bounding boxes, we cite the Faster Bounding Box (FBB) as an example of semi-automatic annotation method [16]. This method was used to generate the Tampere University of Technology (TUT) Indoor dataset. The largest subset is annotated with prediction rectangles, generated by the Faster R-CNN object detector [29], trained on the smallest annotated subset.

2.3. Performance Measure of Detectors Using Rectangles

The evaluation of the effectiveness of the object detector is then performed on the second subset of the annotated dataset. Since we are only dealing here with annotations using bounding boxes, the most used criterion to compare two rectangles is the Jaccard’s similarity index, also known as the Intersection over Union (IoU). For each image of the testing dataset, the IoU measures the percentage of overlap between the prediction bounding rectangle generated by the detector, and the ground truth bounding box , as follows:

We note that the score lies in the interval . The closer this index gets to 1, the better the detection of the object. We say that an object is well detected (or true positive) if the score is greater than or equal to a threshold discussed by experts. In most studies, the threshold value is set at 0.5 [30]. The score causes a problem when its value is zero. This value is embarrassing since it does not explain how far is the prediction bounding rectangle from the ground truth bounding box. To work around this problem, Rezatofighi et al. [30] suggest replacing the with the Generalized Intersection over Union index. The between two rectangles and is defined as follows:where is the smallest convex set enclosing both and .

The exact IoU computation between two HBR is simple. Much software integrates functions to calculate this index. On the other hand, the exact IoU computation between two OBR is not as simple as that between two HBR. Liu et al. (Yao et al., respectively) proposed in Ref. [31] (in [32]) heuristic to estimate the IoU between two OBR. Recently, Zaidi has shown in Ref. [28] that these heuristics give reliable results only when the centers of the rectangles are very close and the angle between the rectangles is small. Moreover, he has developed in Ref. [28] an algorithm that calculates the exact IoU value between two OBR, as well as an estimator of the IoU.

The annotation method has a direct bearing on the performance measurement of an object detector. The OOD reserved for the test phase must be carefully annotated so that the IoU computation between the ground truth and prediction rectangles is not biased.

3. Motivation for the Study

The manual or semi-automatic annotation methods described in Section 2 have made it possible to build many large-scale object datasets. These datasets have given rise to very powerful object detectors. However, we cannot ignore the following concerns:(1)According to paragraph A, accurate manual annotation is expensive, time-consuming, and requires annotation experts.(2)In many situations, the annotator can be in front of an object with an ambiguous orientation as shown in Figure 1. In this case, he will take a considerable time to decide whether the object will be annotated with a horizontal rectangle or an oriented rectangle. Moreover, in both cases, the final decision is not based on any objective argument and depends solely on the dexterity of the annotator.(3)The semi-automatic annotation defined in paragraph B does not guarantee the optimality of the annotation rectangle, in the sense that it must have a minimum area and cover the whole of the object.(4)According to paragraph C, the use of any approximation methods of , whether to measure the accuracy of an oriented object detector or to compare the performance of two oriented object detectors, could lead to biased results. Indeed, the function is strictly increasing over the interval , where and denote the areas of and , respectively. Therefore, a large or a small ground truth bounding rectangle directly induces a bias in the computation of the IoU.

(a)

(b)

(a)
(b)

Figure 1 

(b) Intuitively, it seems that the horizontal rectangle is best suited to annotate the colored object in blue (Figure (a)). The method we propose takes as input the set and returns the oriented rectangle . Since covers the whole object and , then it is natural to annotate the object by the rectangle .

The contribution of this study is the scarcity of annotation methods motivated us to develop a robust semi-automatic annotation method of OOD. Our method is semi-automatic because it consists of a manual step followed by a computer-assisted step. In addition, this method is robust because the bounding rectangle generated by our algorithm is insensitive to the dexterity of the annotator. More precisely, the steps are as follows:(i)We develop an algorithm called RANGE-MBR, which determines from a set of the most relevant (in the sense given by Definition 1) points picked on the object outline, a rectangle enclosing and having a quasi-minimal area, in time.(ii)We propose a new approach to simultaneously build HOD and OOD from a large-scale image bank, based on both the RANGE-MBR algorithm and threshold angles.(iii)We conduct a large experimental study to quantify the performance of the RANGE-MBR algorithm.(iv)We compare the performance of RANGE-MBR to that of the benchmark algorithm, RC-MBR, which determines the minimum rectangle enclosing in time.

This study seems essential to us because it responds to a real need in the development of oriented object detectors. In addition, it allows researchers to build their own oriented object datasets in a rigorous manner.

Definition 1. (relevant point of an object) Any point located on the contour of the object is said to be relevant if it is a local maximum, a local minimum, the most right point, the most left point, a cusp of the first type, a cusp of the second type, an inflection point, and so on.
The remainder of this manuscript is structured as follows. Section 4 deals with the parametrization of rectangles. Section 5 is devoted to the development of the RANGE-MBR algorithm. We explain in Section 6 how to use the RANGE-MBR (or RC-MBR) algorithm to simultaneously generate horizontal and oriented datasets from a large-scale image bank. In Section 7, we perform various numerical experiments to evaluate the performance of the RANGE-MBR algorithm.

4. Parametrization of Bounding Rectangles

First, we give an overview of the different types of parameterization encountered in the literature. Then, we justify the choice of the parameterization associated with the annotation method that we propose in this study. Our choice of parameterization was discussed in a previous study on the accurate computation of the IoU [28]. However, it seems useful to us to recall this choice to facilitate the reading of the manuscript.

4.1. Parameterization of Horizontal Rectangles

The minimum parameterization of a horizontal rectangle requires four parameters. The object dataset PASCAL VOC [18] annotates the rectangle with , where are the coordinates of the vertex , (cf. Figure 2). The object dataset Ms COCO [17] annotates the rectangle with , where and denote the lengths of the line segments and , respectively, and are the coordinates of the vertex (cf. Figure 2). The object dataset ImageNet [19] annotates the rectangle with , where and denote the lengths of the line segments and , respectively, and are the coordinates of the center of (cf. Figure 2).

Figure 2 

Parameterization of horizontal and oriented rectangles.

4.2. Parameterization of Oriented Rectangles

Five parameters—, and —are necessary for the minimal parameterization of an orientated rectangle denoted by the formula ; these are the only parameters that are required. The coordinates for the center of are , which correspond to the coupling and . The lengths of the line segments and are indicated by the parameters and , respectively (cf. Figure 2). The parameter is used to determine the acute angle that exists between the lines and , where denotes the horizontal line that goes through . These two object datasets, HRSC2016 [22] and UCAS-AOD [23], are annotated with the help of this parametrization.

The above -based parametrization is not suitable for annotating satellite image datasets. These OOD have the particularity of containing a large number of instances per image, at different scales, and whose parts overlap [20]. A simple solution that overcomes the drawbacks of the -based annotation consists in defining a rectangle using the coordinates of the vertices , respectively. To solve the indeterminacy problem linked to the permutations of the vertices, we can sort the vertices clockwise and fix the first point according to the following rules [27]:(i)For objects with a distinguished head and a tail (e.g., vehicles and helicopters), the annotator carefully selects to indicate the left corner of the instance head.(ii)For other objects (e.g., tennis courts and bridges), is the point at the top left of the instance.

This type of parametrization was used for the annotation of the DOTA object dataset [20].

4.3. Our Choice of Parametrization

We use the notation to denote an oriented rectangle whose vertices are , , , and . We assume that these vertices are sorted in counterclockwise order, such that is the left vertex with the smallest vertical coordinate.

To sort the vertices set of the rectangle as described above, we first consider the barycenter of the rectangle . Then, we calculate the polar angle of each vertex relative to the barycenter [28]. We denote by , the sorted vertices in ascending order of their polar angle. If , then , else (cf. Figure 2). For more details on the implementation of this sorting method, we refer the reader to our article [28].

In this investigation, we describe a bounding rectangle by using the eight-tuple , where are the coordinates of the -th vertex of the sorted rectangle .

This choice of parameterization is ideally suited to the annotation approach that we suggest since it eliminates the need for extra computations to draw rectangles or to locate the images of rectangles by applying affine transformations. This is one of the reasons why the method is so efficient. In addition, we demonstrate in Section B how to format the outputs of the annotation technique that we propose in accordance with the annotation rule of the DOTA dataset. This is done by referring to the annotation rule.

5. Determination of the Bounding Rectangle

It is natural to annotate the object with the minimum rectangle enclosing the set of relevant points. This problem is formulated as follows:

Problem 1. Given a set of points , we find a rectangle with the smallest area, enclosing . Such a rectangle is called a minimum bounding rectangle of and denoted by MBR .
The solution of problem 5.1 is not unique as shown in Figure 3. So, MBR denotes any solution of Problem 1.
Freeman and Shapira proved in 1975 that one edge of MBR must be collinear with an edge of the convex hull (CH ) of . They proposed in Ref. [33] a natural algorithm to find MBR in time, based on sweeping all minimum rectangles, enclosing , and having an edge collinear with an edge of CH . In 1978, Shamos proposed in Ref. [34] the famous rotating calipers algorithm, which return all pairs of antipodal vertices of a -sided convex polygon in time. In 1983, Toussaint used the rotating calipers technique and developed the algorithm RC-MBR to find MBR in time [35]. In 2006, Dimitrov et al. proposed in Ref. [36, 37] the algorithm PCA-MBR, to approximate MBR , in time, by the minimum bounding rectangle, which is aligned with the eigenvectors of the covariance matrix of CH . They also proved that the relative error between PCA-MBR and MBR is bounded from above by . The main drawback of the algorithm PCA-MBR is that it admits an infinite number of solutions if the covariance matrix of CH has a double eigenvalue. To overcome the problem of nonuniqueness inherent in algorithms RC-MBR and PCA-MBR, we propose the method RANGE-MBR to approximate MBR in time.

Figure 3 

. The green and red rectangles are two distinct solutions of Problem 1.

5.1. The Range-MBR Algorithm

Let be a Cartesian frame of the two-dimensional affine plane. All angles we refer to here are measured counterclockwise from the positive -axis. Let be the line passing through and making an angle with the axis . Let be perpendicular to and passing through . Let and ( and , respectively) be the extreme points of the orthogonal projection of on (, respectively). Let be the minimum bounding rectangle having an edge collinear with . Then, are the intersection points of the two parallel lines to and passing through and , respectively, with the two parallel lines to and passing through and respectively. The area of is given by the product of lengths of the line segments and . From now on, the MBR enclosing and having an edge collinear with the direction making an angle from the positive -axis will be denoted by .

Let , and ) be two unit direction vector of and , respectively. Then, and are two Cartesian frames of the two-dimensional affine plane. For all , we denote by (, respectively) the coordinates of with respect to the frame (, respectively). Let be the rotation matrix of angle , and then , where is the transpose of .

Let be a random variable with mean and standard deviation . Let be observations on the variable . Let be the order statistics of . We define the range of by . Therefore, the lengths of and are given by:

We denote by any estimation of obtained from . On the basis of numerous works carried out on the estimation of the standard deviation from the sample range ([38–43]), we can assume that there is a real constant such that

Besides, it is well known thatwhich is an estimate of , where . It follows that there is a real constant such that:

We assume that the coordinates of the set of points are observations of two random variables and . Since the , then combining equations (6) and (3) gives an approximation of , satisfying the following relation:where . To alleviate notions, we will also use and to designate and . Since , we obtain

Using (8) and some trigonometric identities, we prove thatwhere and

Combining equations (7), (9), and (10) givesThus, the area of MBR is approximately equal toTherefore, we propose to approximate MBR by MBR such that:

The first derivative of with respect to is given by:where

The function has a unique critical point such that

Lemma 1. The second derivative of at the critical point has the same sign as .

Proof. The second derivative of with respect to is given by:Multiplying both sides of the relation (17) by yieldsMultiplying both sides of the relation (14) by yieldsUsing relations (19) and (18) with (i.e., ), we obtain:Therefore, has the same sign as . Since , then what ever the set may be. Thus, has the same sign as .
Therefore, the determination of goes through six cases: Case 1: if , then . Case 2: if and , then is determined through the eigen decomposition of the empirical covariance matrix of . This case is discussed below. Case 3: if and , then . Case 4: if and , then . Case 5: if and , then . Case 6: if and , then it is equivalent to and . This case is discussed below.

5.1.1. Illustrative Example of Case 6

It is difficult to illustrate all the scenarios, which fall into Case 6. The special case of regular polygons is the one that the annotator may encounter when annotating regular objects such as road signs, buildings, and human faces. Moreover, we have pointed out this case to warn the user to pay attention to regularly shaped objects when using our annotation method.

Proposition 1. If the elements of are the vertices of a regular -sided polygon, then and .

Proof. In the general case, the vertices of a regular polygon are uniformly distributed over a circle with radius and center equal to the barycenter of . Thus, without loss of generality, we can assume that , , and the vertex of isConsider the complex sequence , where and is the imaginary unit. Then,Since , then , andSince , thenBesides, is the real part ofThus, . Moreover, using , we deduce thatSince is the imaginary part ofthen .
Note that in the case where the elements of are the vertices of a regular -sided polygon, problem 5.1 has exactly different solutions , . For all , is the image of by the rotation of angle .

5.1.2. Handling of the Exceptional Cases 2 and 6

Cases 2 and 6 are indeterminate cases for which the minimum of the objective function does not exist. This is why we have thought of a trick to get out of the indeterminacy of each case, by returning to the initial definition of the solution.(i)For Case 2, the application of an affine isometry to the set of points makes it possible to migrate to another case for which the angle is well determined. We then obtain the solution of the initial problem by applying the inverse isometry to the solution of the transformed problem, since isometry preserves the areas.(ii)For Case 6, the use of an extra point belonging to the convex hull of the set , allows to migrate to another case for which the angle is well determined. Another alternative consists in asking the user to click on a point of the object’s outline until we get out this state.

Case 2: is negative because of . To get rid of , we have just to follow the same reasoning on a set of pointswhose coordinates and are uncorrelated. Letbe the covariance matrix of the elements of , where . Let be an eigen decomposition of , where (the set of 2-orthogonal matrices), and . For all , set and . Then,twhere . Since , then the linear map defined by preserves the norm and the dot product. Thus, and MBR is the image of MBR under the map .

Proposition 2. If the vectors and are uncorrelated and have different variances , then has a unique minimum at the angle .

Proof. Since and are uncorrelated, then , and and . According to equation (16), the function has a unique critical point . Using Lemma 1, is the maximum of . Case 6. To get out of the indeterminacy of Case 6, we propose to add to the set an artificial point located in the convex hull of so that the empirical covariance matrix of the new set is different from a scalar matrix. We assume that are the Cartesian coordinates of with respect to , where is the barycenter of . Without any loss of generality, we assume that (, respectively) is the vector of the -coordinates (-coordinates, respectively) of the points of with respect to . We denote by and . Lemma 2 gives the relation between and , respectively.