The Eighth Visual Object Tracking VOT2020 Challenge Results

The Visual Object Tracking challenge VOT2020 is the eighth annual tracker benchmarking activity organized by the VOT initiative. Results of 58 trackers are presented; many are state-of-the-art trackers published at major computer vision conferences or in journals in the recent years. The VOT2020 challenge was composed of five sub-challenges focusing on different tracking domains: (i) VOT-ST2020 challenge focused on short-term tracking in RGB, (ii) VOT-RT2020 challenge focused on “real-time” short-term tracking in RGB, (iii) VOT-LT2020 focused on long-term tracking namely coping with target disappearance and reappearance, (iv) VOT-RGBT2020 challenge focused on short-term tracking in RGB and thermal imagery and (v) VOT-RGBD2020 challenge focused on long-term tracking in RGB and depth imagery. Only the VOT-ST2020 datasets were refreshed. A significant novelty is introduction of a new VOT short-term tracking evaluation methodology, and introduction of segmentation ground truth in the VOT-ST2020 challenge – bounding boxes will no longer be used in the VOT-ST challenges. A new VOT Python toolkit that implements all these novelites was introduced. Performance of the tested trackers typically by far exceeds standard baselines. The source code for most of the trackers is publicly available from the VOT page. The dataset, the evaluation kit and the results are publicly available at the challenge website (http://votchallenge.net).

1. 1.

   http://homepages.inf.ed.ac.uk/rbf/CAVIARDATA1.

2. 2.

   http://www.homeoffice.gov.uk/science-research/hosdb/i-lids.

3. 3.

   http://www-sop.inria.fr/orion/ETISEO.

4. 4.

   http://vision.fe.uni-lj.si/cvbase06/.

5. 5.

   http://www.micc.unifi.it/LTDT2014/.

6. 6.

   http://videonet.team.

7. 7.

   http://votchallenge.net.

8. 8.

   http://www.votchallenge.net/vot2020/participation.html.

9. 9.

   http://www.votchallenge.net/vot2019/res/list0_prohibited_1000.txt.

10. 10.

    The target was sought in a window centered at its estimated position in the previous frame. This is the simplest dynamic model that assumes all positions within a search region contain the target have equal prior probability.

11. 11.

    This includes standard FFT-based as well as more recent deep learning based DCFs (e.g., [5, 13]).

12. 12.

    https://github.com/NVIDIA/TensorRT.

13. 13.

    https://github.com/Daikenan/LTMU.

This work was supported in part by the following research programs and projects: Slovenian research agency research programs P2-0214, Z2-1866, P2-0094, Slovenian research agency project J2-8175. Jiři Matas and Ondrej Drbohlav were supported by the Czech Science Foundation Project GACR P103/12/G084. Aleš Leonardis was supported by MURI project financed by MoD/Dstl and EPSRC through EP/N019415/1 grant. Michael Felsberg and Linbo He were supported by WASP, VR (ELLIIT and NCNN), and SSF (SymbiCloud). Roman Pflugfelder and Gustavo Fernández were supported by the AIT Strategic Research Programme 2020 Visual Surveillance and Insight. The challenge was sponsored by the Faculty of Computer Science, University of Ljubljana, Slovenia.

Authors and Affiliations

1. University of Ljubljana, Ljubljana, Slovenia

   Matej Kristan, Luka Čehovin Zajc & Alan Lukežič

2. University of Birmingham, Birmingham, UK

   Aleš Leonardis & Jinyu Yang

3. Czech Technical University, Prague, Czech Republic

   Jiří Matas & Ondrej Drbohlav

4. Linköping University, Linköping, Sweden

   Michael Felsberg, Linbo He, Yushan Zhang, Andreas Robinson & Felix Jaremo Lawin

5. Austrian Institute of Technology, Seibersdorf, Austria

   Roman Pflugfelder & Gustavo Fernández

6. TU Wien, Vienna, Austria

   Roman Pflugfelder

7. Tampere University, Tampere, Finland

   Joni-Kristian Kämäräinen & Song Yan

8. ETH Zürich, Zürich, Switzerland

   Martin Danelljan, Goutam Bhat, Luc Van Gool & Radu Timofte

9. Beijing Institute of Technology, Beijing, China

   Yushan Zhang

10. Carnegie Mellon University, Pittsburgh, USA

    Alexander Hauptmann

11. Dalian University of Technology, Dalian, China

    Bin Yan, Dong Wang, Haojie Zhao, Huchuan Lu, Jianhua Li, Kenan Dai, Lijun Wang, Pengyu Zhang, Shuhao Chen, Yingming Wang & Zezhou Wang

12. Eskisehir Osmangazi University, Eskişehir, Turkey

    Bedirhan Uzun & Hakan Cevikalp

13. Eskisehir Technical University, Eskişehir, Turkey

    Hasan Saribas

14. Five AI, London, UK

    Luca Bertinetto

15. Fuzhou University, Fuzhou, China

    Xiao Ke, Yu Ye & Yuezhou Li

16. High School Affiliated to Renmin University of China, Beijing, China

    Kaicheng Yu

17. Indian Institute of Technology, Tirupati, Tirupati, India

    Hari Chandana Kuchibhotla, Mohana Murali Dasari & Rama Krishna Sai Gorthi

18. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Bing Li, Kaiwen Liu, Weiming Hu & Zhipeng Zhang

19. Jiangnan University, Wuxi, China

    Hui Li, Shaochuan Zhao, Xiao-Jun Wu, Xuefeng Zhu, Yingjie Jiang & Zhangyong Tang

20. KAIST, Daejeon, Korea

    Seokeon Choi

21. Korea University, Seoul, Korea

    Junhyun Lee & Yunsung Lee

22. Megvii, Beijing, China

    Xi Qiu

23. Microsoft Research, Redmond, USA

    Houwen Peng, Jianlong Fu & Zhirong Wu

24. Nanjing University of Information Science and Technology, Nanjing, China

    Kang Yang & Ning Wang

25. National Technical University of Ukraine, Kiev, Ukraine

    Anton Varfolomieiev

26. NLP, Beijing, China

    Qiang Wang & Weiming Hu

27. Remark Holdings, London, UK

    Xiaoyun Yang

28. Samsung Research China-Beijing (SRC-B), Beijing, China

    Jiaqian Yu, Jingtao Xu & Yiwei Chen

29. Sensetime, Taiwan, Hong Kong

    Chen Qian, Fei Wang & Guangqi Chen

30. Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, China

    Shoumeng Qiu, Xiaolin Zhang & Yuzhang Gu

31. Sharif University of Technology, Tehran, Iran

    Shohreh Kasaei

32. Sichuan University, Chengdu, China

    Haoran Bai

33. Southeast University, Nanjing, China

    Fei Xie & Yuncon Yao

34. Stony Brook University, Stony Brook, USA

    Haibin Ling & Heng Fan

35. Tamkang University, New Taipei City, Taiwan

    Chi-Yi Tsai

36. Universidad Autónoma de Madrid, Madrid, Spain

    Álvaro García-Martín, Awet Haileslassie Gebrehiwot & Jesus Bescos

37. University of Alberta, Edmonton, Canada

    Javad Khaghani, Li Cheng & Seyed Mojtaba Marvasti-Zadeh

38. University of Electronic Science and Technology of China, Chengdu, China

    Zhijun Mai

39. University of Isfahan, Isfahan, Iran

    Alireza Memarmoghadam

40. University of Oxford, Oxford, UK

    Li Zhang & Philip H. S. Torr

41. University of Science and Technology of China, Hefei, China

    Houqiang Li, Ning Wang & Wengang Zhou

42. University of Surrey, Guildford, UK

    Josef Kittler, Tianyang Xu & Zhen-Hua Feng

43. University of Udine, Udine, Italy

    Christian Micheloni, Gian Luca Foresti & Matteo Dunnhofer

44. Uppsala University, Uppsala, Sweden

    Fredrik Gustafsson & Thomas B. Schön

45. Yazd University, Yazd, Iran

    Hossein Ghanei-Yakhdan

46. Zhejiang Dahua Technology, Binjiang, China

    Haitao Zhang, Jun Yin, Linyuan Wang, Miao Cheng, Wei Lu & Ziang Ma

Corresponding author

Correspondence to Matej Kristan .

Editors and Affiliations

1. University of Clermont Auvergne, Clermont Ferrand, France

   Adrien Bartoli

2. Università degli Studi di Udine, Udine, Italy

   Andrea Fusiello

A VOT-ST2020 and VOT-RT2020 Submissions

This appendix provides a short summary of trackers considered in the VOT-ST2020 and VOT-RT2020 challenges.

1.1 A.1 Discriminative Sing-Shot Segmentation Tracker (D3S)

A. Lukezic

alan.lukezic@fri.uni-lj.si

Template-based discriminative trackers are currently the dominant tracking paradigm due to their robustness, but are restricted to bounding box tracking and a limited range of transformation models, which reduces their localization accuracy. We propose a discriminative single-shot segmentation tracker named D3S [50], which narrows the gap between visual object tracking and video object segmentation. A single-shot network applies two target models with complementary geometric properties, one invariant to a broad range of transformations, including non-rigid deformations, the other assuming a rigid object to simultaneously achieve high robustness and online target segmentation.

1.2 A.2 Visual Tracking by Means of Deep Reinforcement Learning and an Expert Demonstrator (A3CTDmask)

M. Dunnhofer, G. Foresti, C. Micheloni

{matteo.dunnhofer, gianluca.foresti, christian.micheloni}@uniud.it

A3CTDmask is the combination of the A3CTD tracker [16] with a one-shot segmentation method for target object mask generation. A3CTD is a real-time tracker built on a deep recurrent regression network architecture trained offline using a reinforcement learning based framework. After training, the proposed tracker is capable of producing bounding box estimates through the learned policy or by exploiting the demonstrator. A3CTDmask exploits SiamMask [74] by reinterpreting it as a one-shot segmentation module. The target object mask is generated inside a frame patch obtained through the bounding box estimates given by A3CTD.

1.3 A.3 Deep Convolutional Descriptor Aggregation for Visual Tracking (DCDA)

Y. Li, X. Ke

liyuezhou.cm@gmail.com, kex@fzu.edu.cn

This work aims to mine the target representation capability of pre-trained VGG16 model for visual tracking. Based on spatial and semantic priors, a central attention mask is designed for robust-aware feature aggregation, and an edge attention mask is used for accuracy aware feature aggregation. To make full use of the scene context, a regression loss is developed to learn a discriminative feature for complex scenes. DCDA tracker is implemented based on the Siamese network, with a feature fusion and template enhancement strategies.

1.4 A.4 IOU Guided Siamese Networks for Visual Object Tracking (IGS)

M. Dasari, R. Gorthi

{ee18d001, rkg}@iittp.ac.in

In the proposed IOU-SiamTrack framework, a new block called ‘IOU module’ is introduced. This module accepts the above feature domain response maps, convert them into image domain with the help of anchor boxes, as is done in the inference stage in [41, 42]. Using the classification response map, top-K ‘probable’ bounding boxes, having top-K responses are selected. IOU module then calculates the IOU of probable bounding boxes w.r.t. estimated bounding box and produce the one with maximum IOU score as predicted output bounding box. Through training progress, predicted box is more aligned with ground truth, as network is guided to minimise the IOU loss.

1.5 A.5 SiamMask_SOLO (SiamMask_S)

Y. Jiang, Z. Feng, T. Xu, X. Song

yj.jiang@stu.jiangnan.edu.cn, {z.feng, tianyang.xu}@surrey.ac.uk,

x.song@jiangnan.edu.cn

The SiamMask_SOLO tracker is based on the SiamMask algorithm. It utilizes a multi-layer aggregation module to make full use of different levels of deep CNN features. Besides, to balance all the three branches, the mask branch is replaced by a SOLO [75] head that uses CoordConv and FCN, which improves the performance of the proposed SiamMask_SOLO tracker in terms of both accuracy and robustness. The original refined module is kept for a further performance boost.

1.6 A.6 Diverse Ensemble Tracker (DET50)

N. Wang, W. Zhou, H. Li

wn6149@mail.ustc.edu.cn, {zhwg, lihq}@ustc.edu.cn

In this work, we leverage an ensemble of diverse models to learn manifold representations for robust object tracking. Based on the DiMP method, a shared backbone network (ResNet-50) is applied for feature extraction and multiple head networks for independent predictions. To shrink the representational overlaps among multiple models, both model diversity and response diversity regularization terms are used during training. This ensemble framework is end-to-end trained in a data-driven manner. After box-level prediction, we use SiamMask for mask generation.

1.7 A.7 VPU_SiamM: Robust Template Update Strategy for Efficient Object Tracking (VPU_SiamM)

A. Gebrehiwot, J. Bescos, Á. García-Martín

awet.gebrehiwot@estudiante.uam.es, {j.bescos, alvaro.garcia}@uam.es

The VPU_SiamM tracker is an improved version of the SiamMask [74]. The SiamMask tracks without any target update strategy. In order to enable more discriminant features and to enhance robustness, the VPU_SiamM applies a target template update strategy, which leverages both the initial ground truth template and a supplementary updatable template. The initial template provides highly reliable information and increase robustness against model drift and the updatable template integrates the new target information from the predicted target location given by the current frame. During online tracking, VPU_SiamM applies both forward and backward tracking strategies by updating the updatable target template with the predicted target. The tracking decision on the next frame is determined where both templates yield a high response map (score) in the search region. Data augmentation strategy has been implemented during the training process of the refinement branch to become robust in handling motion-blurred and low-resolution datasets during inference.

1.8 A.8 RPT: Learning Point Set Representation for Siamese Visual Tracking (RPT)

H. Zhang, L. Wang, Z. Ma, W. Lu, J. Yin, M. Cheng

1067166127@qq.com, {wanglinyuan, kobebean, lwhfh01}@zju.edu.cn, {yin_jun, cheng_miao}@dahuatech.com

RPT tracker is formulated with a two-stage structure. The first stage is composed with two parallel subnets, one for target estimation with RepPoints [84] in an offline-trained embedding space, the other trained online to provide high robustness against distractors [13]. The online classification subnet is set to a lightweight 2-layer convolutional neural network. The target estimation head is constructed with Siamese-based feature extraction and matching. For the second stage, the set of RepPoints with highest confidence (i.e. online classification score) is fed into a modified D3S [50] to obtain the segmentation mask. A segmentation map is obtained by combining enhanced target location channel with target and background similarity channels. The backbone is ResNet50 pre-trained on ImageNet, while the target estimation head is trained using pairs of frames from YouTube-Bounding Box [59], COCO [45] and ImageNet VID [63] datasets.

1.9 A.9 Tracking Student and Teacher (TRASTmask)

M. Dunnhofer, G. Foresti, C. Micheloni

{matteo.dunnhofer, gianluca.foresti, christian.micheloni}@uniud.it

TRASTmask is the combination of the TRAST tracker [17] with a one-shot segmentation method for target object mask generation. TRAST tracker consists of two components: (i) a fast processing CNN-based tracker, i.e. the Student; and (ii) an off-the-shelf tracker, i.e. the Teacher. The Student is trained offline based on knowledge distillation and reinforcement learning, where multiple tracking teachers are exploited. Tracker TRASTmask uses DiMP [5] as the Teacher. The target object mask is generated inside a frame patch obtained through the bounding box estimates given by TRAST tracker.

1.10 A.10 Ocean: Object-aware Anchor-free Tracking (Ocean)

Z. Zhang, H. Peng

zhangzhipeng2017@ia.ac.cn, houwen.peng@microsoft.com

We extend our object-aware anchor-free tracking framework [92] with novel transduction and segmentation networks, enabling it to predict accurate target mask. The transduction network is introduced to infuse the knowledge of the given mask in the first frame. Inspired by recent work TVOS [89], it compares the pixel-wise feature similarities between the template and search features, and then transfers the mask of the template to an attention map based on the similarities. We add the attention map to backbone features to learn target-background aware representations. Finally, a U-net shape segmentation pathway is designed to progressively refine the enhanced backbone features to target mask. The code will be completely released at https://github.com/researchmm/TracKit.

1.11 A.11 Tracking by Student FUSing Teachers (TRASFUSTm)

M. Dunnhofer, G. Foresti, C. Micheloni

{matteo.dunnhofer, gianluca.foresti, christian.micheloni}@uniud.it

The tracker TRASFUSTm is the combination of the TRASFUST tracker [17] with a one-shot segmentation method for target object mask generation. TRASFUSTm tracker consists of two components: (i) a fast processing CNN-based tracker, i.e. the Student; (ii) a pool of off-the-shelf trackers, i.e. Teachers. The Student is trained offline based on knowledge distillation and reinforcement learning, where multiple tracking teachers are exploited. After learning, through the learned evaluation method, the Student is capable to select the prediction of the best Teacher of the pool, thus performing robust fusion. Both trackers DiMP [5] and ECO [12] were chosen as Teachers. The target object mask is generated inside a frame patch obtained through the bounding box estimates given by TRASFUSTm tracker.

1.12 A.12 Alpha-Refine (AlphaRef)

B. Yan, D. Wang, H. Lu, X. Yang

yan_bin@mail.dlut.edu.cn, {wdice, lhchuan}@dlut.edu.cn,

xyang@remarkholdings.com

We propose a simple yet powerful two-stage tracker, which consists of a robust base tracker (super-dimp) and an accurate refinement module named Alpha-Refine [82]. In the first stage, super-dimp robustly locates the target, generating an initial bounding box for the target. Then in the second stage, based on this result, Alpha-Refine crops a small search region to predict a high-quality mask for the tracked target. Alpha-Refine exploits pixel-wise correlation for fine feature aggregation, and uses non-local layer to capture global context information. Besides, Alpha-Refine also deploys a delicate mask prediction head [60] to generate high-quality masks. The complete code and trained models of Alpha-Refine will be released at github.com/MasterBin-IIAU/AlphaRefine.

1.13 A.13 Hierarchical Representations with Discriminative Meta-Filters in Dual Path Network for Tracking (DPMT)

F. Xie, N. Wang, K. Yang, Y. Yao

220191672@seu.edu.cn, 20181222016@nuist.edu.cn,

yangkang779@163.con, 220191672@seu.edu.cn

We propose a novel dual path network with discriminative meta-filters and hierarchical representations to solve these issues. DPMT tracker consists of two pathways: (i) Geographical Sensitivity Pathway (GASP) and (ii) Geometrically Sensitivity Pathway (GESP). The modules in Geographical Sensitivity Pathway (GASP) are more sensitive to the spatial location of targets and distractors. Subnetworks in Geometrically Sensitivity Pathway (GESP) are designed to refine the bounding box to fit the target. According to this dual path network design, Geographical Sensitivity Pathway (GASP) should be trained to own more discriminative power between foreground and background while Geographical Sensitivity Pathway (GASP) should focus more on the appearance model of the object.

1.14 A.14 SiamMask (siammask)

Q. Wang, L. Zhang, L. Bertinetto, P.H.S. Torr, W. Hu

qiang.wang@nlpr.ia.ac.cn, {lz, luca}@robots.ox.ac.uk, philip.torr@eng.ox.ac.uk, wmhu@nlpr.ia.ac.cn

Our method, dubbed SiamMask, improves the offline training procedure of popular fully-convolutional Siamese approaches for object tracking by augmenting their loss with a binary segmentation task. In this way, our tracker gains a better instance-level understanding towards the object to track by exploiting the rich object mask representations offline. Once trained, SiamMask solely relies on a single bounding box initialisation and operates online, producing class-agnostic object segmentation masks and rotated bounding boxes. Code is publicly available at https://github.com/foolwood/SiamMask.

1.15 A.15 OceanPlus: Online Object-Aware Anchor-Free Tracking (OceanPlus)

Z. Zhang, H. Peng, Z. Wu, K. Liu, J. Fu, B. Li, W. Hu

zhangzhipeng2017@ia.ac.cn, houwen.peng@microsoft.com,

Wu.Zhirong@microsoft.com, liukaiwen2019@ia.ac.cn, jianf@microsoft.com,

bli@nlpr.ia.ac.cn, wmhu@nlpr.ia.ac.cn

This model is the extension of the Ocean tracker A.10. Inspired by recent online models, we introduce an online branch to accommodate to the changes of object scale and position. Specifically, the online branch inherits the structure and parameters from the first three stages of the Siamese backbone network. The fourth stage keeps the same structure as the original ResNet50, but its initial parameters are obtained through the pre-training strategy proposed in [5]. The segmentation refinement pathway is the same as Ocean. We refer the readers to Ocean tracker A.10 and https://github.com/researchmm/TracKit for more details.

1.16 A.16 fastOcean: Fast Object-Aware Anchor-Free Tracking (fastOcean)

Z. Zhang, H. Peng

zhangzhipeng2017@ia.ac.cn, houwen.peng@microsoft.com

To speed up the inference of our submitted tracker OceanPlus, we use TensorRT^{Footnote 12} to re-implement the model. All structure and model parameters are the same as OceanPlus. Please refer to OceanPlus A.15 and Ocean A.10 for more details.

1.17 A.17 Siamese Tracker with Discriminative Feature Embedding and Mask Prediction (SiamMargin)

G. Chen, F. Wang, C. Qian

{chenguangqi, wangfei, qianchen}@sensetime.com

SiamMargin is based on the SiamRPN++ algorithm [41]. In the training stage, a discrimination loss is added to the embedding layer. In the training phase the discriminative embedding is offline learned. In the inference stage the template feature of the object in current frame is obtained by ROIAlign from features of the current search region and it is updated via a moving average strategy. The discriminative embedding features are leveraged to accommodate the appearance change with properly online updating. Lastly, the SiamMask [74] model is appended to obtain the pixel-level mask prediction.

1.18 A.18 Siamese Tracker with Enhanced Template and Generalized Mask Generator (SiamEM)

Y. Li, Y. Ye, X. Ke

liyuezhou.cm@gmail.com, yyfzu@foxmail.com, kex@fzu.edu.cn

SiamEM is a Siamese tracker with enhanced template and generalized mask generator. SiamEM improves SiamFC++ [81] by obtaining feature results of the template and flip template in the network header while making decisions based on quality scores to predict bounding boxes. The segmentation network presented in [10] is used as a mask generation network.

1.19 A.19 TRacker by Using ATtention (TRAT)

H. Saribas, H. Cevikalp, B. Uzun

{hasansaribas48, hakan.cevikalp, eee.bedirhan}@gmail.com

The tracker ‘TRacker by using ATtention’ uses a two-stream network which consists of a 2D-CNN and a 3D-CNN, to use both spatial and temporal information in video streams. To obtain temporal (motion) information, 3D-CNN is fed by stacking the previous 4 frames with one stride. To extract spatial information, the 2D-CNN is used. Then, we fuse the two-stream network outputs by using an attention module. We use ATOM [13] tracker and ResNet backbone as a baseline. Code is available at https://github.com/Hasan4825/TRAT.

1.20 A.20 InfoGAN Based Tracker: InfoVITAL (InfoVital)

H. Kuchibhotla, M. Dasari, R. Gorthi

{ee18m009, ee18d001, rkg}@iittp.ac.in

Architecture of InfoGAN (Generator, Discriminator and a Q-Network) is incorporated in the Tracking-By-Detection Framework using the Mutual Information concept to bind two distributions (latent code) to the target and the background samples. Additional Q Network helps in proper estimation of the assigned distributions and the network is trained offline in an adversarial fashion. During online testing, the additional information from the Q-Network is used to obtain the target location in the subsequent frames. This greatly helps to assess the drift from the exact target location from frame-to-frame and also during occlusion.

1.21 A.21 Learning Discriminative Model Prediction for Tracking (DiMP)

G. Bhat, M. Danelljan, L. Van Gool, R. Timofte

{goutam.bhat, martin.danelljan, vangool, timofter}@vision.ee.ethz.ch

DiMP is an end-to-end tracking architecture, capable of fully exploiting both target and background appearance information for target model prediction. The target model here constitutes the weights of a convolution layer which performs the target-background classification. The weights of this convolution layer are predicted by the target model prediction network, which is derived from a discriminative learning loss by applying an iterative optimization procedure. The model prediction network employs a steepest descent based methodology that computes an optimal step length in each iteration to provide fast convergence. The online learned target model is applied in each frame to perform target-background classification. The final bounding box is then estimated using the overlap maximization approach as in [13]. See [5] for more details about the tracker.

1.22 A.22 SuperDiMP (SuperDiMP)

G. Bhat, M. Danelljan, F. Gustafsson, T. B. Schön, L. Van Gool, R. Timofte

{goutam.bhat, martin.danelljan}@vision.ee.ethz.ch, {fredrik.gustafsson,

thomas.schon}@it.uu.se, {vangool, timofter}@vision.ee.ethz.ch

SuperDiMP [23] combines the standard DiMP classifier from [5] with the EBM-based bounding-box regressor from [14, 22]. Instead of training the bounding box regression network to predict the IoU with an \(L_2\) loss [5], it is trained using the NCE+ approach [23] to minimize the negative-log likelihood. Further, the tracker uses better training and inference settings.

1.23 A.23 Learning What to Learn for Video Object Segmentation (LWTL)

G. Bhat, F. Jaremo Lawin, M. Danelljan, A. Robinson, M. Felsberg, L. Van Gool, R. Timofte

goutam.bhat@vision.ee.ethz.ch, felix.jaremo-lawin@liu.se,

martin.danelljan@vision.ee.ethz.ch, {andreas.robinson, michael.felsberg}@liu.se, {vangool, timofter}@vision.ee.ethz.ch

LWTL is an end-to-end trainable video object segmentation VOS architecture which captures the current target object information in a compact parametric model. It integrates a differentiable few-shot learner module, which predicts the target model parameters using the first frame annotation. The learner is designed to explicitly optimize an error between target model prediction and a ground truth label, which ensures a powerful model of the target object. Given a new frame, the target model predicts an intermediate representation of the target mask, which is input to the offline trained segmentation decoder to generate the final segmentation mask. LWTL learns the ground-truth labels used by the few-shot learner to train the target model. Furthermore, a network module is trained to predict spatial importance weights for different elements in the few-shot learning loss. All modules in the architecture are trained end-to-end by maximizing segmentation accuracy on annotated VOS videos. See [7] for more details.

1.24 A.24 Adaptive Failure-Aware Tracker (AFAT)

T. Xu, S. Zhao, Z. Feng, X. Wu, J. Kittler

tianyang.xu@surrey.ac.uk, zsc960813@163.com, z.feng@surrey.ac.uk,

wu_xiaojun@jiangnan.edu.cn, j.kittler@surrey.ac.uk

Adaptive Failure-Aware Tracker [80] is based on Siamese structure. First, multi-RPN module is employed to predict the central location with Resnet-50. Second, a 2-cell LSTM is established to perform quality prediction with an additional motion model. Third, fused mask branch is exploited for segmentation.

1.25 A.25 Ensemble Correlation Filter Tracking Based on Temporal Confidence Learning (TCLCF)

C. Tsai

chiyi_tsai@gms.tku.edu.tw

TCLCF is a real-time ensemble correlation filter tracker based on the temporal confidence learning method. In the current implementation, we use four different correlation filters to collaboratively track the same target. The TCLCF tracker is a fast and robust generic object tracker without GPU acceleration. Therefore, it can be implemented on the embedded platform with limited computing resources.

1.26 A.26 AFOD: Adaptive Focused Discriminative Segmentation Tracker (AFOD)

Y. Chen, J. Xu, J. Yu

{yiwei.chen, jingtao.xu, jiaqian.yu}@samsung.com

The proposed tracker is based on D3S and DiMP [5], employing ResNet-50 as backbone. AFOD calculates the feature similarity to foreground and background of the template as proposed in D3S. For discriminative features, AFOD updates the target model online. AFOD adaptively utilizes different strategies during tracking to update the scale of search region and to adjust the prediction. Moreover, the Lovasz hinge loss metric is used to learn the IoU score in offline training. The segmentation module is trained using both databases YoutubeVOS2019 and DAVIS2016. The offline training process includes two stages: (i) BCE loss is used for optimization and (ii) the Lovasz hinge is applied for further fine tuning. For inference, two ResNet-50 models are used; one for the segmentation and another for the target.

1.27 A.27 Fast Saliency-Guided Continuous Correlation Filter-Based Tracker (FSC2F)

A. Memarmoghadam

a.memarmoghadam@yahoo.com

The tracker FSC2F is based on the ECOhc approach [12]. A fast spatio temporal saliency map is added using the PQFT approach [21]. The PQFT model utilizes intensity, colour, and motion features for quaternion representation of the search image context around the previously pose of the tracked object. Therefore, attentional regions in the coarse saliency map can constrain target confidence peaks. Moreover, a faster scale estimation algorithm is utilised by enhancing the fast fDSST method [15] via jointly learning of the sparsely-sampled scale spaces.

1.28 A.28 Adaptive Visual Tracking and Instance Segmentation (DESTINE)

S.M. Marvasti-Zadeh, J. Khaghani, L. Cheng, H. Ghanei-Yakhdan, S. Kasaei

mojtaba.marvasti@ualberta.ca, khaghani@ualberta.ca, lcheng5@ualberta.ca,

hghaneiy@yazd.ac.ir, kasaei@sharif.edu

DESTINE is a two-stage method consisting of an axis-aligned bounding box estimation and mask prediction, respectively. First, DiMP50 [5] is used as the baseline tracker switching to ATOM [13] when IoU and normalized L1-distance between the results meet predefined thresholds. Then, to segment the estimated bounding box, the segmentation network of FRTM-VOS [60] uses the predicted mask by SiamMask [74] as its scores. Finally, DESTINE selects the best target mask according to the ratio of foreground pixels for two predictions. The codes are publicly released at https://github.com/MMarvasti/DESTINE.

1.29 A.29 Scale Adaptive Mean-Shift Tracker (ASMS)

Submitted by VOT Committee

The mean-shift tracker optimizes the Hellinger distance between template histogram and target candidate in the image. This optimization is done by a gradient descend. ASMS [73] addresses the problem of scale adaptation and presents a novel theoretically justified scale estimation mechanism which relies solely on the mean-shift procedure for the Hellinger distance. ASMS also introduces two improvements of the mean-shift tracker that make the scale estimation more robust in the presence of background clutter – a novel histogram colour weighting and a forward-backward consistency check. Code available at https://github.com/vojirt/asms.

1.30 A.30 ATOM: Accurate Tracking by Overlap Maximization (ATOM)

Submitted by VOT Committee

ATOM separates the tracking problem into two sub-tasks: i) target classification, where the aim is to robustly distinguish the target from the background; and ii) target estimation, where an accurate bounding box for the target is determined. Target classification is performed by training a discriminative classifier online. Target estimation is performed by an overlap maximization approach where a network module is trained offline to predict the overlap between the target object and a bounding box estimate, conditioned on the target appearance in first frame. See [13] for more details.

1.31 A.31 Discriminative Correlation Filter with Channel and Spatial Reliability - C++ (CSRpp)

Submitted by VOT Committee

The CSRpp tracker is the C++ implementation of the Discriminative Correlation Filter with Channel and Spatial Reliability (CSR-DCF) tracker [47].

1.32 A.32 Incremental Learning for Robust Visual Tracking (IVT)

Submitted by VOT Committee

The idea of the IVT tracker [62] is to incrementally learn a low-dimensional sub-space representation, adapting on-line to changes in the appearance of the target. The model update, based on incremental algorithms for principal component analysis, includes two features: a method for correctly updating the sample mean, and a forgetting factor to ensure less modelling power is expended fitting older observations.

1.33 A.33 Kernelized Correlation Filter (KCF)

Submitted by VOT Committee

This tracker is a C++ implementation of Kernelized Correlation Filter [24] operating on simple HOG features and Colour Names. The KCF tracker is equivalent to a Kernel Ridge Regression trained with thousands of sample patches around the object at different translations. It implements multi-thread multi-scale support, sub-cell peak estimation and replacing the model update by linear interpolation with a more robust update scheme. Code available at https://github.com/vojirt/kcf.

1.34 A.34 Multiple Instance Learning tracker (MIL)

Submitted by VOT Committee

MIL tracker [1] uses a tracking-by-detection approach, more specifically Multiple Instance Learning instead of traditional supervised learning methods and shows improved robustness to inaccuracies of the tracker and to incorrectly labelled training samples.

1.35 A.35 Robust Siamese Fully Convolutional Tracker (RSiamFC)

Submitted by VOT Committee

RSiamFC tracker is an extended SiamFC tracker [4] with a robust training method which puts a transformation on training sample to generate a pair of samples for feature extraction.

1.36 A.36 VOS SOTA Method (STM)

Submitted by VOT Committee

Please see the original paper for details [56].

1.37 A.37 (UPDT)

Submitted by VOT Committee

Please see the original paper for details [6].

B VOT-LT2020 Submissions

This appendix provides a short summary of trackers considered in the VOT-LT2020 challenge.

1.1 B.1 Long-Term Visual Tracking with Assistant Global Instance Search (Megtrack)

Z. Mai, H. Bai, K. Yu, X. QIu

marchihjun@gmail.com, 522184271@qq.com, valjean1832@outlook.com,

qiuxi@megvii.com

Megtrack tracker applies a 2-stage method that consists of local tracking and multi-level search. The local tracker is based on ATOM [13] algorithm improved by initializing online correlation filters with backbone feature maps and by inserting a bounding box calibration branch in the target estimation module. SiamMask [74] is cascaded to further refining the bounding box after locating the centre of the target. The multi-level search uses RPN-based regression network to generate candidate proposals before applying GlobalTrack [26]. Appearance scores are calculated using both the online-learned RTMDNet [29] and the offline-learned one-shot matching module and linearly combine them to leverage the former’s high robustness and the latter’s discriminative power. Using a pre-defined threshold, the highest-scored proposal is considered as the current tracker state and used to re-initialize the local tracker for consecutive tracking.

1.2 B.2 Skimming-Perusal Long-Term Tracker (SPLT)

B. Yan, H. Zhao, D. Wang, H. Lu, X. Yang

{yan_bin, haojie_zhao}@mail.dlut.edu.cn, {wdice, lhchuan}@dlut.edu.cn,

xyang@remarkholdings.com

This is the original SPLT tracker [83] without modification. SPLT consists of a perusal module and a skimming module. The perusal module aims at obtaining precise bounding boxes and determining the target’s state in a local search region. The skimming module is designed to quickly filter out most unreliable search windows, speeding up the whole pipeline.

1.3 B.3 A Baseline Long-Term Tracker with Meta-Updater (LTMU_B)

K. Dai, D. Wang, J. Li, H. Lu, X. Yang

dkn2014@mail.dlut.edu.cn, {wdice, jianhual}@dlut.edu.cn, lhchuan@dlut.edu.cn,

xyang@remarkholdings.com

The tracker LTMU_B is a simplified version of LTMU [11] and LTDSE with comparable performance adding a RPN-based regression network, a sliding-window based re-detection module and a complex mechanism for updating models and target re-localization. The short-term tracker LTMU_B contains two components. One is for target localization and based on DiMP algorithm [5] using ResNet50 as the backbone network. The update of DiMP is controlled by meta-updater which is proposed by LTMU^{Footnote 13}. The second component is the SiamMask network [74] used for refining the bounding box after locating the centre of the target. It also takes the local search region as the input and outputs the tight bounding boxes of candidate proposals. For the verifier, we adopts MDNet network [5] which uses VGGM as the backbone and is pre-trained on ILSVRC VID dataset. The classification score is finally obtained by sending the tracking result’s feature to three fully connected layers. GlobalTrack [26] is utilised as the global detector.

1.4 B.4 Robust Long-Term Object Tracking via Improved Discriminative Model Prediction (RLTDiMP)

S. Choi, J. Lee, Y. Lee, A. Hauptmann

seokeon@kaist.ac.kr, {ljhyun33, swack9751}@korea.ac.kr, alex@cs.cmu.edu

We propose an improved Discriminative Model Prediction method for robust long-term tracking based on a pre-trained short-term tracker. The baseline tracker is SuperDiMP which combines the bounding-box regressor of PrDiMP [14] with the standard DiMP [5] classifier. To make our model more discriminative and robust, we introduce uncertainty reduction using random erasing, background augmentation for more discriminative feature learning, and random search with spatio-temporal constraints. Code available at https://github.com/bismex/RLT-DIMP.

1.5 B.5 Long-Term MDNet (ltMDNet)

H. Fan, H. Ling

{hefan, hling}@cs.stonybrook.edu

We designate a long-term tracker by adapting MDNet [55]. In specific, we utilize an instance-aware detector [26] to generate target proposals. Then, these proposals are forwarded to MDNet for classification. Since the detector performs detection on the full image, the final tracker can locate the target in the whole image which can robustly deal with full occlusion and out-of-view. The instance-aware detector is implemented by on Faster R-CNN using ResNet-50. The MDNet is implemented as in the original paper.

1.6 B.6 (CLGS)

Submitted by VOT Committee

In this work, we develop a complementary local-global search (CLGS) framework to conduct robust long-term tracking, which is a local robust tracker based on SiamMask [74], a global detection based on cascade R-CNN [8], and an online verifier based on Real-time MDNet [29]. During online tracking, the SiamMask model locates the target in local region and estimates the size of the target according to the predicted mask. The online verifier is used to judge whether the target is found or lost. Once the target is lost, a global R-CNN detector (without class prediction) is used to generate region proposals on the whole image. Then, the online verifier will find the target from region proposals again. Besides, we design an effective online update strategy to improve the discrimination of the verifier.

1.7 B.7 (LT_DSE)

Submitted by VOT Committee

This algorithm divides each long-term sequence into several short episodes and tracks the target in each episode using short-term tracking techniques. Whether the target is visible or not is judged by the outputs from the short-term local tracker and the classification-based verifier updated online. If the target disappears, the image-wide re-detection will be conducted and output the possible location and size of the target. Based on these, the tracker crops the local search region that may include the target and sends it to the RPN based regression network. Then, the candidate proposals from the regression network will be scored by the online learned verifier. If the candidate with the maximum score is above the pre-defined threshold, the tracker will regard it as the target and re-initialize the short-term components. Finally, the tracker conducts short-term tracking until the target disappears again.

1.8 B.8 (SiamDW_LT)

Submitted by VOT Committee

SiamDW_LT is a long-term tracker that utilizes deeper and wider backbone networks with fast online model updates. The basic tracking module is a short-term Siamese tracker, which returns confidence scores to indicate the tracking reliability. When the Siamese tracker is uncertain on its tracking accuracy, an online correction module is triggered to refine the results. When the Siamese tracker is failed, a global re-detection module is activated to search the target in the images. Moreover, object disappearance and occlusion are also detected by the tracking confidence. In addition, we introduce model ensemble to further improve the tracking accuracy and robustness.

C VOT-RGBT2020 Submissions

This appendix provides a short summary of trackers considered in the VOT-RGBT2020 challenge.

1.1 C.1 Multi-model Continuous Correlation Filter for RGBT Visual Object Tracking (M2C2Frgbt)

A. Memarmoghadam

a.memarmoghadam@yahoo.com

Inspired by ECO tracker [12], we propose a robust yet efficient tracker namely as M2C2Frgbt that utilizes multiple models of the tracked object and estimates its position every frame by weighted cumulative fusion of their respective regressors in a ridge regression optimization problem [51]. Moreover, to accelerate tracking performance, we propose a faster scale estimation method in which the target scale filter is jointly learned via sparsely sampled scale spaces constructed by just the thermal infrared data. Our scale estimation approach enhances the running speed of fDSST [15] as the baseline algorithm better than 20% while maintaining the tracking performance as well. To suppress unwanted samples mostly belong to the occlusion or other non-object data, we conservatively update every model on-the-fly in a non-uniform sparse manner.

1.2 C.2 Jointly Modelling Motion and Appearance Cues for Robust RGB-T Tracking (JMMAC)

P. Zhang, S. Chen, D. Wang, H. Lu, X. Yang

pyzhang@mail.dlut.edu.cn, shuhaochn@mail.dlut.edu.cn, wdice@dlut.edu.cn,

lhchuan@dlut.edu.cn, xyang@remarkholdings.com

Our tracker is based on [88], consisting of two components, i.e. multimodal fusion for appearance trackers and camera motion estimation. In multimodal fusion, we develop a late fusion method to infer the fusion weight maps of both RGB and thermal (T) modalities. The fusion weights are determined by using offline-trained global and local Multimodal Fusion Networks (MFNet), and then adopted to linearly combine the response maps of RGB and T modalities obtained from ECOs. In MFNet, the truncated VGG-M networks is used as backbone to extract deep feature. In camera motion estimation, when the drastic camera motion is detected, we compensate movement to correct the search region by key-point-based image registration technique. Finally, we employ YOLOv2 to refine the bounding box. The scale estimation and model updating methods are borrowed from ECO in default.

1.3 C.3 Accurate Multimodal Fusion for RGB-T Object Tracking (AMF)

P. Zhang, S. Chen, B. Yan, D. Wang, H. Lu, X. Yang

{pyzhang, shuhaochn, yan_bin}@mail.dlut.edu.cn, {wdice, lhchuan}@dlut.edu.cn, xyang@remarkholdings.com

We achieve multimodal fusion for RGB-T tracking by linear combining the response maps obtained from two monomodality base trackers, i.e., DiMP. The fusion weight is obtained by the Multimodal Fusion Network proposed in [88]. To achieve high accuracy, the bounding box obtained from fused DiMP is then refined by a refinement module in visible modality. The refinement module, namely Alpha-Refine, aggregates features via a pixel-level correlation layer and a non-local layer and adaptively selects the most adequate results from three branches, namely bounding box, corner and mask heads, which can predict more accurate bounding boxes. Note that the target scale estimated by IoUNet in DiMP is also applied in visible modality which is followed by Alpha-Refine and the model updating method is borrowed from DiMP in default.

1.4 C.4 SqueezeNet Based Discriminative Correlation Filter Tracker (SNDCFT)

A. Varfolomieiev

a.varfolomieiev@kpi.ua

The tracker uses FHOG and convolutional features extracted from both video and infrared modalities. As the convolutional features, the output of the ‘fire2/concat’ layer of the original SqueezeNet network [27] is used (no additional pre-training for the network is performed). The core of the tracker is the spatially regularized discriminative correlation filter, which is calculated using the ADMM optimizer. The calculation of the DCF filter is performed independently over different feature modalities. The filter is updated in each frame using simple exponential forgetting.

1.5 C.5 Decision Fusion Adaptive Tracker (DFAT)

H. Li, Z. Tang, T. Xu, X. Zhu, X. Wu, J. Kittler

hui_li_jnu@163.com, 1030415519@vip.jiangnan.edu.cn, tianyang.xu@surrey.ac.uk, xuefeng_zhu95@163.com, wu_xiaojun@jiangnan.edu.cn, j.kittler@surrey.ac.uk

Decision Fusion Adaptive Tracker is based on Siamese structure. Firstly, the multi-layer deep features are extracted by Resnet-50. Then, multi-RPN module is employed to predict the central location with multi-layer deep features. Finally, an adaptive weight strategy for decision level fusion is utilized to generate the final result. In addition, the template features are updated by a linear template update strategy.

1.6 C.6 Multi-modal Fusion for End-to-End RGB-T Tracking (mfDiMP)

Submitted by VOT Committee

The mfDiMP tracker contains an end-to-end tracking framework for fusing the RGB and TIR modalities in RGB-T tracking [87]. The mfDiMP tracker fuses modalities at the feature level on both the IoU predictor and the model predictor of DiMP [87] and won the VOT-RGBT2019 challenge.

1.7 C.7 Online Deeper and Wider Siamese Networks for RGBT Visual Tracking (SiamDW-T)

Submitted by VOT Committee

SiamDW-T is based on previous work by Zhang and Peng [91], and extends it with two fusion strategies for RGBT tracking. A simple fully connected layer is appended to classify each fused feature to background or foreground. SiamDW-T achieved the second rank in VOT-RGBT2019 and its code is available at https://github.com/researchmm/VOT2019.

D VOT-RGBD2020 Submissions

This appendix provides a short summary of trackers considered in the VOT-RGBD2020 challenge.

1.1 D.1 Accurate Tracking by Category-Agnostic Instance Segmentation for RGBD Image (ATCAIS)

Y. Wang, L. Wang, D. Wang, H. Lu, X. Yang

{wym097,wlj,wdice,lhchuan}@dlut.edu.cn, xyang@remarkholdings.com

The proposed tracker combines both instance segmentation and the depth information for accurate tracking. ATCAIS is based on the ATOM tracker and the HTC instance segmentation method which is re-trained in a category-agnostic manner. The instance segmentation results are used to detect background distractors and to re-fine the target bounding boxes to prevent drifting. The depth value is used to detect the target occlusion or disappearance and re-finding the target.

1.2 D.2 Depth Enhanced DiMP for RGBD Tracking (DDiMP)

S. Qiu, Y. Gu, X. Zhang

{shoumeng, gyz, xlzhang}@mail.sim.ac.cn

DDiMP is based on SuperDiMP which combines the standard DiMP classifier from [5] with the bounding box regressor from [5]. The update strategy of the model during the tracking process is enhanced by using the model’s confidence for the current tracking results. Output of IoU-Net is used to determine whether to fine-tune the shape, size, and position of the target. To handle scale variations, the target is searched over five scales \(1.025^{\{-2, -1, 0, 1, 2\}}\), and depth information is utilized to prevent scale from changing too quickly. Finally, two trackers with different model update confidence thresholds run in parallel, and the output with higher confidence is selected as the tracking result of the current frame.

1.3 D.3 Complementary Local-Global Search for RGBD Visual Tracking (CLGS-D)

H. Zhao, Z. Wang, B. Yan. D. Wang, H. Lu, X. Yang

{haojie_zhao,zzwang,yan_bin,wdice,lhchuan@dlut.edu.cn}@mail.dlut.edu.cn,

xyang@remarkholdings.com

CLGS-D tracker is based on SiamMask, FlowNetv2, CenterNet, Real-time MDNet and a novel box refine module. The SiamMask model is used as the base tracker. MDNet is used to judge whether the target is found or lost. Once the target is lost, CenterNet is used to generate region proposals on the whole image. FlowNetv2 is used to estimate the motion of the target by generating a flow map. Then, the region proposals are filtered with aid of the flow and depth maps. Finally, an online “verifier” will find the target from the remaining region proposals again. A novel module is also used in this work to refine the bounding box.

1.4 D.4 Siamese Network for Long-term RGB-D Tracking (Siam_LTD)

X.-F. Zhu, H. Li, S. Zhao, T. Xu, X.-J. Wu

{xuefeng_zhu95,hui_li_jnu,zsc960813,wu_xiaojun}@163.com,

tianyang.xu@surrey.ac.uk

Siam_LTD employes ResNet-50 to extract backbone features and RPN branch to locate the centre. In addition, a re-detection mechanism is introduced.

Reprints and permissions

© 2020 Springer Nature Switzerland AG

Policies and ethics