UnifiedFace: A Uniform Margin Loss Function for Face Recognition

Abstract

Face recognition has achieved great success due to the development of deep convolutional neural networks (DCNNs) and loss functions based on margin. However, complex DCNNs bring a large number of parameters as well as computational effort, which pose a significant challenge to resource-constrained embedded devices. Meanwhile, the popular margin-based loss functions all introduce only one type of margin and cannot further introduce a larger margin to achieve tighter classification boundary. In contrast to the common approach, we believe that additive and multiplicative margins should be used jointly to introduce larger margins from the margin perspective. Therefore, we propose a new margin-based loss function called UnifiedFace. First, we introduce an additive margin in the target angle activation function. Second, we add a multiplicative margin in the non-target angle. UnifiedFace introduces both additive and multiplicative margins, allowing for the introduction of large margins to achieve more compact intra-class variance and closer separated inter-class variance. In addition, we specifically design efficient face recognition models called GhostFaceNet for resource-constrained embedded devices. Experimental results demonstrate that UnifiedFace achieves state-of-the-art performance or performance competed with popular methods in training datasets of different sizes. UnifiedFace achieves optimal performance in models of varying complexity. Moreover, competitive results are achieved in the large-scale test set IJBB/C, especially the state-of-the-art performance achieved in TAR ($FAR=1e-6$). GhostFaceNet can significantly improve operational efficiency without significantly degrading recognition performance, making it ideal for embedded devices with limited resources.

1. Introduction

In recent years, face recognition has achieved remarkable success due to the rapid development of deep convolutional neural networks(DCNNs) [1,2,3] and loss functions, especially the loss function based on margin [4,5,6,7,8,9]. Moreover, face recognition technology has been widely used in the fields of finance, public security, 3D sensors in face recognition applications [10], and Crowd counting models adapted to drone-assisted systems [11]. However, complex DCNNs generate a large number of parameters and computational consumption, which is not friendly to resource-limited embedded or mobile devices. Meanwhile, these loss functions embed the face features into the hypersphere space and further increase the intra-class compactness and inter-class separation by introducing margins. Nevertheless, all these margin-based loss functions introduce only one type of margin and cannot further introduce larger margins to achieve tighter classification bounds.

To alleviate the problem of the excessive number of parameters and computational effort associated with complex DCCNs, a series of methods have been proposed to build efficient and lightweight convolutional neural networks, such as knowledge distillation [12,13], networks pruning [14,15], and quantization [16]. In addition, recent research has focused on building efficient convolutional networks with fewer parameters and computational effort and has achieved great success. Several works [17,18,19] propose the construction of efficient modules to design lightweight convolutional neural networks. Depthwise separable convolution operations and inverse residual structures are used to construct efficient modules in MobileNet [17,20,21] and obtain competitive performance. To achieve state-of-the-art performance, channel shuffle operation is utilized in ShuffleNet [18]. Although these methods achieve lightweight convolutional neural networks with low computational cost and a low number of parameters by designing efficient units, they do not take redundant information in features into account. For example, as shown in Figure 1a,b, we mark similar features with the same color. GhostNet [19] employs ordinary convolution to generate intrinsic features, followed by inexpensive linear transformations to produce ghost features. The ghost feature reveals sufficient information about the intrinsic features and is able to reduce convolutional consumption with similar recognition performance. Nevertheless, the direct use of the ghost module in face recognition can lead to a significant degradation in model performance. We, therefore, reconstructed an efficient model called GhostFaceNet, based on the ghost module, which significantly reduces the number of parameters and the amount of computation without noticeably degrading performance.

Margin-based softmax loss functions are proposed to further reduce intra-class compactness and increase inter-class dispersion. The margin-based loss function usually embeds the face features into the hypersphere space and introduces large margins. The core idea is to redesign the target activation function $cos\left({\theta }_y\right)$ in the softmax cross-entropy loss function. ${\theta }_y$ denotes the angle between the sample x and the weight $W_y$, where y is the true label of the sample x. The redesigned target activation function needs to be guaranteed to be monotonically decreasing in $[0,\pi ]$, so that the angle ${\theta }_y$ can become smaller when $cos\left({\theta }_y\right)$ reaches the same value. The large margins include additive and multiplicative margins. SphereFace [5] and SphereFaceR [7] further enhance intra-class compactness and inter-class separation by introducing multiplicative margins. CosFace [6,8] and ArcFace [9] achieve the same goal by introducing additive angle margins. Formally, the multiplicative margin is multiplied directly on angle ${\theta }_y$, and the additive margin is added to angle ${\theta }_y$ or acts on $cos\left({\theta }_y\right)$. Furthermore, both approaches have achieved significant performance. Multiplicative and additive margins share a common goal of improving intra-class compactness and inter-class separability by introducing large margins to make decision boundaries tighter. Intuitively, larger margins and stricter decision bounds result by introducing both multiplicative and additive margins.

To effectively unify the additive and multiplicative margins, we propose a unified margin loss function called UnifiedFace. Specifically, the additive and multiplicative margins jointly influence ${\theta }_y$ so that it becomes smaller while reaching the same value as $cos\left({\theta }_y\right)$. We introduce a multiplicative margin in the non-target angle activation function and an additive margin in the target angle activation function. As far as we know, this is the first time that both additive and multiplicative margins have been introduced in the loss function with this idea. Meanwhile, the additive and multiplicative margins are no longer independent but are related. We refer to this approach as UnifiedFace.

Our contributions are summarized as follows:

(1) We propose a new efficient and lightweight model called GhostFaceNet based on ghost module, which significantly reduces the number of parameters and the amount of computation without noticeably degrading performance.

(2) We propose the frame of a unified margin by introducing both additive and multiplicative margins. This frame can effectively introduce larger margins and achieve tighter decision bounds. Moreover, the additive and multiplicative margins are no longer independent of each other but are related.

(3) In this framework, we design a new margin-based loss function called UnifiedFace. UnifiedFace introduces a multiplicative margin in the non-target angle activation function and an additive margin in the target angle activation function. UnifiedFace achieves more stable training and superior generalization. Meanwhile, we give a clear geometric interpretation of UnifiedFace.

The structure of the paper is as follows. Section 2 briefly reviews the work related to deep face recognition and lightweight networks frameworks. Section 3 describes our proposed approach in detail, including the GhostFaceNet and UnifiedFace. Experimental results and related discussions are given in Section 4 and the conclusion is presented in Section 5.

2. Related Works

2.1. Lightweight Networks Frameworks

The efficient networks structure is an important guarantee for achieving high-performance face recognition. We mainly focus on lightweight networks structures and lightweight face recognition algorithms proposed in recent years.

MobileNetV1 [20] proposes to replace the ordinary convolution with depthwise separable convolution for the first time, which significantly reduces the computational effort and ensures the performance of the model at the same time. In addition, MobileNetV2 [17] designs the inverse residual structure and linear bottleneck layer to make the model structure more reasonable and efficient. MobileNetV3 [21] utilizes complementary search techniques to combine these modules into more efficient models. Group convolutions and channel confusion operations are used in ShuffleNet [18,22] to communicate information from different channels. Ref. [23] uses basic convolution and pooling operations to build a lightweight face detection model and successfully applies it to the corresponding system. The system also performs very well in hazardous situations such as underwater and in avalanches. GhostNet [19] employs ordinary convolution to generate intrinsic features, followed by inexpensive linear transformations to produce ghost features. The ghost feature reveals sufficient information about the intrinsic features and can reduce convolutional consumption with similar recognition performance.

Although excellent results have been achieved with lightweight models, there are no lightweight convolutional neural networks specifically designed for face recognition, especially for resource-constrained embedded devices. MobileFaceNet [24] and ShuffleFaceNet [25] are designed based on MobileNetV2 and ShuffleNetV2, respectively, and achieve significant accuracy. MobileFaceNet designs a global depthwise convolution layer and replaces global average pooling. The use of efficient convolution of variable groups and equivalent angular distillation loss functions in VarGFaceNet [26] makes the model more discriminative and explanatory. MobiFace [27] builds the model using a residual bottleneck layer with an extended layer and aims to maximize the information in the output vector. It achieves an excellent performance of 99.73% in the LFW dataset. ELANet [2] adds an efficient attention module to MobileNetV2 and applies a feature fusion strategy to significantly improve the performance of lightweight models across postures and ages at a very small computational cost.

2.2. Deep Face Recognition

Significant advances in deep face recognition are due to the rapid development of deep convolutional neural networks [28,29,30]. Early studies consider open-set face recognition to be essentially a multi-classification problem [28,29]. DeepID2 [31] elaborately designs convolutional neural networks(CNNs) and trains CNNs with softmax loss and contrast loss to achieve excellent performance. FaceNet [32] uses triplet loss for training and learns the mapping of face images to Euclidean space directly. Furthermore, it has achieved state-of-the-art results in large-scale face recognition tasks. Center loss [3] enhances the discriminative ability of deep learning features by penalizing the distance between deep features and their corresponding class centers and substantially improves the performance. Other methods based on contrastive loss and triplet loss, such as [33,34], achieve excellent performance and illustrate the importance of metric learning in open set validation.

Margin-based loss function is the main research direction of deep face recognition in recent years and has been widely popular due to its simplicity and effectiveness. L-softmax [4] introduces large angular margins in deep feature learning, and subsequent margin-based loss functions follow this design idea. SphereFace [5] introduces multiplicative margins to achieve tighter decision boundary and introduces weight normalization. To learn large angular margin face embedding features, [35,36,37] introduce feature normalization. SphereFace does not fully implement the multiplicative margin intuition within $[0,\pi ]$ and its training process is unstable. Therefore, SphereFaceR [7] proposes a simpler and more efficient approach. SphereFaceR can apply multiplicative margins in any of the ranges $[0,\pi ]$, and SphereFaceR v2 applies margins in non-target angle activation functions for the first time. CosFace [6,8] and ArcFace [9] achieve impressive results by using additive margins instead of multiplicative margins. Specifically, CosFace adds the cosine margin penalty directly to the target logit, and ArcFace adds an angular margin penalty to supervise the network training. Although margin-based loss functions have achieved great success in the field of face recognition, intuitively, introducing both multiplicative and additive margins is more conducive to network training and obtaining superior performance. Other margin-based loss functions achieved significant performance, such as Circle loss [38], Adacos [39], AdaptiveFace [40], CurricularFace [41], and DiscFace [42].

3. Our Method

In this section, we first describe our proposed GhostFaceNet-Bottleneck (GF-Bottleneck). The lightweight network framework called GhostFaceNet built from the GF-Bottleneck module is presented in detail. Secondly, we review popular margin-based loss functions in recent years, including additive and multiplicative margins. Finally, we propose a uniform margin method called UnifiedFace and give its clear geometric interpretation.

3.1. GhostFaceNet

Based on ghost module, we propose the GhostFaceNet with the configuration shown in

Table 1. The overall framework of GhostFaceNet. n represents the number of module rereads, and s indicates the number of similar features generated by each intrinsic feature.

Input	Operator	n	s	Output
$112\times 112$	Conv $3\times 3$	-	2	64
$56\times 56$	GF-Bottleneck	1	2	64
$28\times 28$	GF-Bottleneck	4	1	64
$28\times 28$	GF-Bottleneck	1	2	128
$14\times 14$	GF-Bottleneck	5	1	128
$14\times 14$	GF-Bottleneck	1	2	256
$7\times 7$	GF-Bottleneck	4	1	256
$7\times 7$	Conv $1\times 1$	-	1	1024
$7\times 7$	GDC	-	1	1024
$1\times 1$	Linear $1\times 1$	-	1	128

. GhostFaceNet mainly consists of a stack of GhostFaceNet-Bottleneck (GF-Bottleneck).

In the ghost module, the input features are first convolved by ordinary convolution to generate a fixed number $c^{{}^{\prime }}$ of intrinsic features $Y\in {\mathbb{R}}^{w^{{}^{\prime }}\times h^{{}^{\prime }}\times c^{{}^{\prime }}}$:

$$\begin{array}{c}\hfill Y=X\ast f\end{array}$$

(1)

where $X\in {\mathbb{R}}^{w\times h\times c}$ is the input feature and $f\in {\mathbb{R}}^{k\times k\times c\times c^{{}^{\prime }}}$ is the convolution kernel. The bias terms are ignored for simplicity. Then, the intrinsic features generate ghost features $y_{ij}$, i.e., features associated with the intrinsic features, by some series of linear operations (LO):

$$\begin{array}{c}\hfill y_{ij}={\varphi }_{ij}(y_i^{{}^{\prime }}),i=1,2,\cdots ,m,j=1,2,\cdots ,s\end{array}$$

(2)

where $y_{ij}$ is the i-th feature in intrinsic feature Y and ${\varphi }_{ij}$ is the linear operation that generates the j-th associated feature. m is the number of generated intrinsic features, and s represents the number of ghost features generated for each intrinsic feature. The mathematical relationship between m and s is as follows:

$$\begin{array}{c}\hfill n=m\times s\end{array}$$

(3)

where n is the number of channels of the final output feature. Feature information $y_{ij}$ and intrinsic features are connected together to output the final feature information. The operation flow of the ghost module is shown in Figure 2.

GF-Bottleneck mainly consists of ghost modules, depthwise separable convolutions, and $1\times 1$ convolutions, as shown in Figure 3. The input features $X\in {\mathbb{R}}^{w\times h\times c}$ are first divided into two branches by normal convolution in DG module. A branch consists of LO in DG modules, a depthwise separable convolution with a step size of 1, and $1\times 1$ convolution. The other one acts as an identity. We define the output ratio input as the output expansion ratio as follows:

$$\begin{array}{c}\hfill r=\frac{n}{c}\end{array}$$

(4)

According to ShuffleNetV2 [18], we set $r=1$ for all modules, to reduce memory access cost(MAC). The feature information of the two branches is connected by the concat operation and finally output by the channel shuffle operation to allow the communication of the different channel features.

We construct a new structure to increase the number of output features as well as spatial downsampling, as illustrated in Figure 3b. Unlike the previous case with stride = 1, in the identity branch, we add the depthwise separable convolution and $1\times 1$ convolution to enrich feature expression. In addition, we set stride = 2.

Based on GF-Bottleneck, we propose the GhostFaceNet with the configuration shown in Table 1. GhostFaceNet mainly consists of s stack of GF-Bottleneck.

The first layer of the network is a standard $3\times 3$ convolution used for fast downsampling. A series of GF-Bottleneck modules are applied, and the number of feature channels is gradually increased. Global average pooling (GAP) treats each unit of the output features with the same importance. However, it contradicts the theory that different units contribute differently to the face feature vector. Refs. [9,43] show that GAP is inaccurate for face recognition tasks. The global depthwise convolution (GDC) in [20] assigns different weights to units with various degrees of importance. Meanwhile, the fully connected (FC) layer generates more weight information and thus increases the size of the model. In GhostFaceNet, GDC is used instead of GAP. PReLu [44] is chosen to be nonlinear to replace ReLU [45]. Finally, a linear $1\times 1$ convolution is used to output compact 128-dimensional face features. GhostFaceNet has 0.82 M parameters and 0.15 GFLOPs, far less than the parameters and computations of other lightweight networks. Meanwhile, GhostFaceNet can well verify the performance of UnifiedFace in lightweight networks.

3.2. Margin-Based Loss Functions

The traditional softmax loss function, which does not explicitly encourage enhanced intra-class compactness and inter-class separation, is expressed as follows:

$$\begin{array}{c}\hfill L_{softmax}=-\frac{1}{N}\sum _i^{N}log(\frac{e^{W_y^T{x}_i+b_y}}{{\sum }_j^n{e}^{W_j^T{x}_i+b_j}})\end{array}$$

(5)

where N is batch size and n is the number of classes. $b_y$ indicates the bias term for the y-th class.

To effectively alleviate this problem, margin-based softmax loss functions are proposed with great success. Commonly, the margin-based softmax constraint redefines logit as the following equation:

$$\begin{array}{c}\hfill W_j^T{x}_i=\Vert W_j\Vert \Vert x_i\Vert cos{\theta }_j\end{array}$$

(6)

where ${\theta }_j$ denotes the angle between the feature and the j-th weight vector. The feature is normalized by $L_2$ normalization and then re-specified as s. The weight is fixed to 1 by $L_2$ normalization.

In [7], $cos\theta$ is replaced by the target angle activation function (non-target angle activation function). The margin-based softmax is uniformly defined as follows

$$\begin{array}{c}\hfill L=log(1+\sum _{i\ne y}{e}^{Q\left({\theta }_y,{\theta }_i,s,m\right)})\end{array}$$

(7)

$$\begin{array}{c}\hfill Q\left({\theta }_y,{\theta }_i,s,m\right)=s·(\eta \left({\theta }_i\right)-\eta \left({\theta }_y\right)+\Delta \left({\theta }_y\right)\end{array}$$

(8)

The m indicates the margin, including additive margin or multiplicative margin. $\eta \left({\theta }_i\right)-\eta \left({\theta }_y\right)$ denotes the confidence difference of various classes. $\Delta \left(\theta \right)$ is the characteristic function to describe angular margins, including the size of the margin and the training stability. The characteristic function is defined as follows:

$$\begin{array}{c}\hfill \Delta \left(\theta \right)=\eta \left(\theta \right)-\psi \left(\theta \right)>0\end{array}$$

(9)

where $\eta \left(\theta \right)$ and $\psi \left(\theta \right)$ are defined as the target activation function and non-target activation function, respectively. In particular, different $\eta \left(\theta \right)$ and $\psi \left(\theta \right)$ constitute various margin-based loss functions, as shown in

Table 2. Different margin−based loss functions for the characteristic functions.

Method	Non-Target Function	Target Function	Characteristic Function
SphereFace	$cos\left(\theta \right)$	${(-1)}^{k}cos\left(m\theta \right)-2k$	$cos\left(\theta \right)-{(-1)}^{k}cos\left(m\theta \right)+2k$
SphereFaceR v1	$cos\left(\theta \right)$	$cos(min(m,\frac{\pi }{\theta })·\theta )$	$cos(\theta )-cos(min(m,\frac{\pi }{\theta })·\theta )$
SphereFaceR v2	$cos(\frac{\theta }{m})$	$cos\left(\theta \right)$	$cos(\frac{\theta }{m})-cos\left(\theta \right)$
CosFace	$cos\left(\theta \right)$	$cos\left(\theta \right)-m$	m
ArcFace	$cos\left(\theta \right)$	$cos(\theta +m)$	$cos\left(\theta \right)-cos(\theta +m)$
UnifiedFace	$cos(\frac{\theta +m}{M})$	$cos(\theta +m)$	$cos(\frac{\theta +m}{M})-cos(\theta +m)$

. A comparison of the different margin-based loss functions for the characteristic functions is given in Figure 4a. Despite the great progress of these margin-based loss functions, the problem of how to introduce margins more efficiently remains a tricky one. That is, designing a simple and excellent performance angular margin is still a problem. It is natural to think of unifying additive and multiplicative margins to solve this problem.

3.3. UnifiedFace

To unify the additive and multiplicative margins, an additive margin is introduced in the target angle activation function $\psi \left(\theta \right)$ and a multiplicative margin is added in the non-target angle activation function $\eta \left(\theta \right)$. Following the conventional additive margin loss function [9], we first designed $\psi \left(\theta \right)$, as follows:

$$\begin{array}{c}\hfill \psi \left(\theta \right)=cos(\theta +m)\end{array}$$

(10)

where m is usually an additive margin penalty.

In addition, to design a reasonable loss function with multiplicative margin, we construct $\psi \left(\theta \right)$ and $\eta \left(\theta \right)$ satisfying the following relation:

$$\begin{array}{c}\hfill \psi \left(\theta \right)=\eta \left(M\theta \right),\theta \in [0,\pi ]\end{array}$$

(11)

where M is usually the specified normal number. Therefore, we design a novel non-target activation function $\eta \left(\theta \right)$:

$$\begin{array}{c}\hfill \eta \left(\theta \right)=cos(\frac{\theta +m}{M})\end{array}$$

(12)

where $M=e^m$.

From Figure 4b, we can observe the change in the trend of UnifiedFace at different m in detail. The characteristic functions at different values of m all follow the angular margin minimum for simple samples and maximum for medium hard samples. Meanwhile, a sufficiently large angular margin is given for hard samples. Compared with other popular loss functions of characteristic functions, UnifiedFace can obtain the maximum angular margin for medium hardness samples. Furthermore, UnifiedFace does not give too much angular margin to hard samples leading to an excessive focus on these hard samples during training compare to SphereFaceR v2.

To facilitate understanding, the binary case is considered to illustrate intuitively the uniform additive and multiplicative margins. Figure 5 intuitively compares the no angular margin, multiplicative angular margin, additive angular margin, and uniform angular margin. For a sample x, if no margin is introduced, $cos\left({\theta }_1\right)=cos\left({\theta }_2\right)$ becomes the decision boundary for class 1, i.e., the decision boundary is ${\theta }_1={\theta }_2$, as illustrated in Figure 5a. When both additive and multiplicative margins are introduced, it also means that $\psi \left(\theta \right)={\psi }_1\left(\theta \right)$ and $\eta \left(\theta \right)={\eta }_1\left(\theta \right)$. $cos({\theta }_1+m)=cos\left(\frac{{\theta }_2+m}{M}\right)$ is the decision boundary for class 1. When the sample x belongs to class 1, it is required that $cos({\theta }_1+m)>cos\left(\frac{{\theta }_2+m}{M}\right)$. Correctly classifying sample x into class 1 requires ${\theta }_1+m<\frac{{\theta }_2+m}{M}$, while correctly classifying sample x into class 2 requires ${\theta }_2+m<\frac{{\theta }_1+m}{M}$. As shown in

Table 3. Comparison of binary bounds for different loss functions.

Method	Binary Boundary for Class 1
softmax	${\theta }_1<{\theta }_2$
SphereFace	$m{\theta }_1<{\theta }_2$
CosFace	$cos\left({\theta }_1\right)-m<{\theta }_2$
ArcFace	${\theta }_1+m<{\theta }_2$
UnifiedFace	${\theta }_1+m<\frac{{\theta }_2+m}{M}$

, the decision boundary of UnifiedFace is more stringent. That is when the sample x belonging to class 1 is correctly classified, ${\theta }_1$ is smaller than the other loss functions of both additive margin and multiplicative margin. The process of UnifiedFace supervise face recognition network is illustrated in Figure 6.

4. Experiments

In this section, we first introduce the dataset used for the experiment, the network structure, and the detailed setup of the experiment. In addition, we give a detailed description of the network structure. Secondly, we conducted ablation studies on different training datasets separately to investigate the effect of their hyperparameters on UnifiedFace. Then, we give the results of comparing UnifiedFace with other state-of-the-art methods. Meanwhile, we report the performance comparison results of UnifiedFace with popular loss functions in networks of different complexity. Finally, we evaluate and compare with state-of-the-art methods in the popular large-scale test IJBB/C.

4.1. Datasets

(1) Training datasets: We select VGGFace2 [46] and MS1M V3 as the training dataset. To illustrate the generalization performance of our proposed method, we selected the large-scale dataset MS1M V3 and the small-scale dataset VGGFace2 for model training, respectively. VGGFace2 includes 3.14M images of faces in different poses, ages, and ethnicities. The 5.1 million face images of 93K identities are included in MS1MV3, which is a cleaned Ms-celeb-1m dataset [47]. No data enhancement is used in the training process.

(2) Validation and test datasets: We choose LFW [48], CPLFW [49], CALFW [50], CFP [51], VGGFace2-FP [46], IJB-B [52], and IJB-C [53] datasets to validate and evaluate our method.

Table 4. Statistical information on the datasets used.

Dataset	ID	Images
MS1M-RetinaFace	93 K	5.1 M
VGGFace2	8.6 K	3.1 M
LFW	5749	13,233
CPLFW	5749	12,174
CALFW	5749	11,652
CFP	500	7000
VGGFace2-FP	500	173 K
IJBB	1845	76.8 K
IJBC	3531	148.8 K

shows the statistics of the datasets.

4.2. Implementation

We use PyTorch [54] to implement all the experiments. We use GhostFaceNet as the backbone network to investigate the performance of the proposed new efficient lightweight model in using UnifiedFace. In addition, we use ResNet-100 [30] as the backbone network. The outputs of ResNet-100 are all 512 dimensions for a fair comparison. The batch size is set to 256. A value of 0.1 is used as the initial learning rate and divided by 10 every 10 epochs. The training ends at 25 epochs. The momentum and the weight decay are set to 0.9 and $5e-5$, respectively. In all experiments, we compare the accuracy of the loss function in different datasets.

4.3. Ablation Experiments

4.3.1. Effect of Hyperparameters M Furthermore, S

We use VGGFace2 and MS1M V3 as training datasets to explore the effect of hyperparameters on the performance of UnifiedFace in training datasets of different scales, respectively.

We first investigate the advantages and disadvantages of the performance of different hyperparameters in the small-scale training dataset VGGFace2, and the experimental results are shown in

Table 5. The ablation study on different hyperparameters s and m with ResNet-100. The VGGFace2 dataset, which is relatively small in size, is chosen for the training dataset.

s	m	LFW	CALFW	CPLFW	VGG2_FP	CFP
						CFP_FF	CFP_FP
64	0.5	99.55%	93.08%	91.92%	95.14%	99.34%	97.47%
64	0.45	99.52%	92.80%	92.23%	94.80%	99.26%	97.01%
64	0.4	99.73%	93.77%	92.58%	95.36%	99.37%	97.83%
40	0.35	99.67%	93.10%	92.48%	95.76%	99.26%	97.59%

. The hyperparameter s is set to 64 and 40, and the hyperparameter m is fixed from 0.35 to 0.5. Experimental results show that UnifiedFace performs well for values of m ranging from 0.35 to 0.4 in small training datasets. In particular, when $s=64$ and $m=0.4$, UnifiedFace achieves the optimal performance. Specifically, the optimal setting for the additive margin of UnifiedFace is between 0.35 and 0.4. The optimal setting for the multiplicative margin of UnifiedFace is between 1.4 and 1.5.

Then, we report the effect of hyperparameters on performance in the large-scale training dataset MS1M V3, and the experimental results are presented in

Table 6. The ablation study on different hyperparameters s and m with ResNet-100. The large-scale MS1M V3 dataset is chosen as the training dataset.

s	m	LFW	CALFW	CPLFW	VGG2_FP	CFP
						CFP_FF	CFP_FP
64	0.5	99.77%	96.00%	92.42%	95.00%	99.77%	97.09%
64	0.45	99.80%	96.00%	92.68%	95.34%	99.77%	97.30%
50	0.4	99.80%	96.17%	92.65%	96.52%	99.79%	97.77%
40	0.35	99.78%	95.90%	92.53%	96.02%	99.76%	97.83%

. The hyperparameters s are set to 64 and 40, and the hyperparameters m are fixed from 0.35 to 0.5. In particular, the model performance is optimal when $s=50$, $m=0.4$. The experimental results show that the optimal setting interval for the hyperparameter m is from 0.4 to 0.45, which means that the optimal setting range for the additive margin is from 0.4 to 0.45, and the optimal setting range for the multiplicative margin is from 1.5 to 1.6. The results of hyperparametric experiments on training datasets of different sizes show that UnifiedFace has excellent generalization ability.

In addition, we give the variation of loss values during UnifiedFace training with different hyperparameters, as shown in Figure 7a,e. During the training process, different hyperparameters lead to different values of the final convergence of the loss values. $s=64$, $m=0.5$, the loss values fluctuate the most. In other cases, the loss value starts with a sharp drop and then falls steadily. Although the loss value fluctuation during the training process is smoothest and converges to a minimum value when m is taken as small as possible, the performance is not the best. We select the best-performing hyperparameters as the experimental parameters and compare the change of loss values during the training process with the popular loss functions in recent years, as illustrated in Figure 7b,f.

4.3.2. The Comparison with State-of-the-Art Methods

The performance comparisons of UnifiedFace with other state-of-the-art methods at different scales of training datasets are given in

Table 7. Performance comparison of UnifiedFace with state-of-the-art methods in using VGGFace2 as the training dataset.

Method	s	m	LFW	CALFW	CPLFW	VGG2_FP	CFP
							CFP_FF	CFP_FP
SphereFace	-	4	99.43%	91.18%	90.33%	95.46%	98.90%	96.99%
SphereFaceR v1	64	1.5	99.63%	93.70%	92.50%	95.08%	99.46%	97.79%
SphereFaceR v2	64	1.5	99.65%	93.37%	92.25%	95.30%	99.37%	97.50%
CosFace	64	0.45	99.63%	93.83%	92.37%	94.40%	99.37%	97.10%
ArcFace	64	0.45	99.62%	93.77%	92.33%	94.90%	99.40%	97.53%
CurricularFace	64	0.5	99.62%	93.90%	91.58%	94.46%	99.43%	96.90%
UnifiedFace	64	0.4	99.73%	93.77%	92.58%	95.36%	99.37%	97.83%

and

Table 8. Performance comparison of UnifiedFace with state-of-the-art methods in using MS1M V3 as the training dataset.

Method	s	m	LFW	CALFW	CPLFW	VGG2_FP	CFP
							CFP_FF	CFP_FP
SphereFace	-	3	99.80%	96.00%	92.42%	95.34%	99.73%	97.57%
SphereFaceR v1	64	1.5	99.78%	96.03%	92.55%	95.04%	99.78%	97.39%
SphereFaceR v2	64	1.5	99.82%	96.03%	92.97%	95.69%	99.77%	98.00%
CosFace	64	0.45	99.82%	96.08%	92.27%	94.38%	99.76%	96.89%
ArcFace	64	0.45	99.77%	96.12%	92.43%	94.66%	99.76%	97.13%
CurricularFace	64	0.5	99.80%	96.02%	92.05%	94.06%	99.77%	96.77%
UnifiedFace	50	0.4	99.80%	96.17%	92.65%	96.52%	99.79%	97.77%

, respectively. UnifiedFace outperforms other state-of-the-art methods when using small-scale datasets as training datasets. Specifically, UnifiedFace achieves state-of-the-art performance in the validation datasets LFW, CPLFW, and CFP_FP. The performance of UnifiedFace improves over the optimal loss function by 0.08%, 0.08%, and 0.04% in these three validation datasets, respectively. Competitive performance is also achieved in the remaining three validation datasets. UnifiedFace also outperforms the current popular loss function when using MS1M V3 as the training dataset. Specifically, UnifiedFace achieves state-of-the-art performance in CALFW, VGG2_FP, and CFP_FF. UnifiedFace substantially improves the performance of the model in the VGG2_FP validation dataset. Furthermore, UnifiedFace achieves suboptimal performance in all other validation sets. A comparison of UnifiedFace with popular loss functions in recent years in training sets of different sizes illustrates its excellent generalization ability.

In addition, the comparison of UnifiedFace with the popular loss function in recent years in terms of loss value change during training is given. Both Figure 7b,f show that UnifiedFace has the most stable loss value change process and the smallest convergence value during the training process. Meanwhile, there is no large and drastic loss change during the training process. The accuracy comparison of UnifiedFace with popular loss functions in LFW and CFP_FP validation datasets during training is shown in Figure 7c,h, respectively.

4.4. Experiments in GhostFaceNet

We report on the performance of GhostFaceNet trained using UnifiedFace., as shown in

Table 9. Performance comparison between UnifiedFace and popular loss functions in lightweight networks.

Loss	m	LFW	CALFW	CFP-FP
SphereFace	4	99.47%	94.97%	91.79%
CosFace	0.3	99.35%	94.27%	92.23%
ArcFace	0.15	99.53%	94.58%	94.21%
ArcFace	0.3	99.52%	94.63%	93.46%
ArcFace	0.35	99.50%	94.67%	93.10%
AirFace	0.45	99.48%	94.98%	92.73%
UnifiedFace	0.45	99.60%	94.53%	92.74%
UnifiedFace	0.4	99.63%	94.80%	93.70%
UnifiedFace	0.35	99.50%	94.63%	94.54%

. Simultaneously, the performance comparison of UnifiedFace with popular loss functions in lightweight networks is given.

The popular loss functions of recent years are used to train GhostFaceNet and compare it with UnifiedFace. The margins of SphereFace, CosFace, and AirFace [55] are set to 4, 0.3, and 0.45, respectively. AirFace is a loss function specifically designed for lightweight network models, and its linear target logit makes the training process very stable. The margins of ArcFace are given as 0.15, 0.30, and 0.35, and are pre-trained by softmax before using ArcFace. On LFW and CFP_FP, UnifiedFace is better than the other loss functions. In particular, UnifiedFace substantially improves the performance of the model in the cross-pose verification set CFP_FP. Furthermore, in CALFW, UnifiedFace achieves similarly competitive results. The experimental results also demonstrate the degradation of model performance caused by excessive margins in lightweight networks when using margin-based loss functions for supervised training. For example, the performance for the case of m = 0.45 is lower than that of m = 0.35 in UnifiedFace. Likewise, the performance in ArcFace is similar. The fact that UnifiedFace still achieves optimal performance in lightweight networks despite the introduction of large margins illustrates the advantage of introducing both additive and multiplicative margins in UnifiedFace.

4.5. Comparison Experiments with Different Lightweight Network Models

We compared the performance of GhostFaceNet with popular models in recent years, including the complex model ResNet-50 and other popular lightweight models. We used UnifiedFace as the loss function.

The performance of GhostFaceNet is shown in

Table 10. Performance comparison of GhostFaceNet with other models. Results are in % and higher number indicates better performance.

Model	m	LFW	CALFW	CFP-FP	PARAM	FLOPs
ResNet-50	0.45	99.61%	94.81%	95.17%	40.25 M	2.19 G
EfficientNet	0.45	99.52%	95.31%	96.55%	6.58 M	1.14 G
MobileNetV2	0.45	99.55%	94.87%	93.40%	2.26 M	0.43 G
MobileFaceNet	0.45	99.53%	94.95%	95.49%	1.0 M	0.45 G
GhostNet	0.45	99.58%	94.78%	95.40%	4.06 M	0.44 G
GhostFaceNet	0.45	99.60%	94.53%	92.74%	0.82 M	0.15 G
GhostFaceNet	0.4	99.63%	94.80%	93.70%	0.82 M	0.15 G
GhostFaceNet	0.35	99.50%	94.63%	94.54%	0.82 M	0.15 G

. Although GhostFaceNet achieves the best performance in the validation dataset LFW, it does not perform well in the cross-pose and cross-age validation datasets. GhostFaceNet does not have a large enough number of channels to learn richer features. Nevertheless, GhostFaceNet generates the lowest number of parameters and computation, especially since it consumes only 0.15 G of computation. Although GhostFaceNet does not perform well in the cross-pose and cross-age validation datasets, its overall performance is still competitive. Meanwhile, it is very computationally small and can significantly reduce the inference time of the model in resource-constrained devices.

4.6. Experiments Evaluated on IJBB/C

In this section, we evaluate the performance of UnifiedFace on popular large-scale benchmarks (i.e., IJB-B and IJB-C). A fair comparison with state-of-the-art methods is provided. Specifically, we train UnifiedFace on MS1M V3. We compare UnifiedFace with the current state-of-the-art approaches in recent years, i.e., SphereFace [5], SphereFaceR [7], CosFace [6,8], ArcFace [9], and CurricularFace [41].

The experimental results in

Table 11. Evaluation on IJB-B and IJB-C. We use ResNet-100 as the backbone architecture and MS1MV3 as the training dataset for all the compared methods. Results are in % and higher number indicates better performance.

Method	m	IJB-B			IJB-C
		$1\times e^{-6}$	$1\times e^{-5}$	$1\times e^{-4}$	$1\times e^{-6}$	$1\times e^{-5}$	$1\times e^{-4}$
SphereFace	3	42.00	85.84	93.90	78.77	92.18	95.68
SphereFaceR v1	1.5	34.56	88.88	94.47	88.39	93.69	95.97
SphereFaceR v2	1.5	37.22	88.45	94.51	87.79	93.34	95.99
CosFace	0.45	41.29	90.21	94.63	89.61	94.02	96.11
ArcFace	0.45	37.41	89.82	94.20	89.42	93.38	95.85
CurricularFace	0.45	36.38	90.06	94.37	89.58	93.83	95.85
UnifiedFace(MS1M V3)	0.4	42.51	88.52	94.03	88.50	93.12	95.62

show that the experimental results in the large-scale test datasets are consistent with those in the validation datasets. This consistency also indicates the validity of the selected test datasets and models. It is also well demonstrated that the models that obtained good performance in the validation datasets can also achieve correspondingly good results in the test datasets. From Table 11, we can observe that although different types of margins produce similar performance, result in different results at a low false acceptance rate (FAR). UnifiedFace can significantly improve the performance on the IJBB and IJBC test datasets. In particular, at TAR ($@FAR=1e-6$), UnifiedFace achieves state-of-the-art performance. Moreover, UnifiedFace performs particularly well at other low FAR, which is of great interest for designing robust face recognition. The receiver operating characteristic curves(ROC) of UnifiedFace with other loss functions are given in Figure 8a,b, respectively.

4.7. Experiments in Embedded Devices

We conducted a 1:1 speed test on a computer platform (i9-12900K CPU). In addition, we also performed accuracy test experiments on the computer platform with the LFW dataset, as illustrated in

Table 12. Lightweight model performance testing in computer platform.

Method	Accuracy	Speed
ResNet-50	99.64%	-
EfficientNet	99.53%	36.5 ms
MobileNetV2	99.55%	25.67 ms
MobileFaceNet	99.53%	23.67 ms
GhostFaceNet	99.63%	15.3 ms

. GhostFaceNet achieve the best performer in the LFW and is 15.3ms faster than EfficientNet, MobileNetV2, and MobileFaceNet, 2.39 times, 1.67 times, and 1.55 times, respectively.

Meanwhile, we conduct running speed comparison experiments in resource-constrained embedded device to further illustrate the advantages of GhostFaceNet in embedded devices. The embedded platform of choice is the ZYNQ 7020, which uses ARM+FPGA SOC technology to integrate ARM Cortex-A9 and FPGA programmable logic on a single chip. We use MTCNN [56] for face images detection and crop it to $112\times 112$. We use NCNN for PyTorch model transformation and implement the final runtime test. We do not apply quantization to the model, so there is no degradation in the model recognition capability.

The runtime results of GhostFaceNet and other methods for single-face image recognition in embedded devices are shown in

Table 13. Speed comparison of GhostFaceNet with different models for single image recognition in embedded devices.

Method	Embedded Platform	PARAM	FLOPs
ResNet-50	3.09 s	40.29 M	2.19 G
EfficientNet	2.4 s	6.58 M	1.14 G
MobileNetV2	0.77 s	2.26 M	0.43 G
MobileFaceNet	0.79 s	1.0 M	0.45 G
GhostFaceNet	0.28 s	0.82 M	0.15 G

. GhostFaceNet is 11.04 times, 8.57 times, 2.75 times, and 2.82 times faster than ResNet50, EfficientNet, MobileNetV2, and MobileFaceNet, respectively. In general, GhostFaceNet outperforms other lightweight models in embedded devices in terms of speed and is suitable for deployment in resource-limited embedded devices. For example, in practical scenarios where system resources are limited, such as time and attendance card punching, community access control, and surveillance devices, it becomes very difficult to deploy complex models, and our proposed GhostFaceNet can perform the relevant recognition tasks very easily.

5. Conclusions

We propose an efficient lightweight convolutional neural network for embedded or mobile devices, called GhostFaceNet, that drastically reduces the amount of computation required by the model. In addition, our paper proposes a unified framework. In this framework, the intuition of introducing larger margins is effectively achieved by introducing both additive and multiplicative margins. We have conducted extensive experiments on many validation datasets and popular benchmark tests to verify the effectiveness of our proposed algorithm. Furthermore, during the actual training process and system testing, we found that our proposed loss function can introduce larger margins without causing training instability, as well as GhostFaceNet is able to effectively reduce the inference time of the model in edge devices. In future work, we will also design more efficient lightweight modules from the perspective of hardware system resources and take advantage of the highly parallel nature of FPGAs to further accelerate the inference speed of the models in edge devices. Moreover, we continue to explore the design of more efficient margins.