A deep learning framework for multi-object tracking in team sports videos

• Main Authors: Cao, Wei, Wang, Xiaoyong, Liu, Xianxiang, Xu, Yishuai
• Format: Article
• Published: Institution of Engineering and Technology (IET) 2024
• Subjects: QA75 Electronic computers. Computer science
• Online Access: http://eprints.um.edu.my/47061/; https://doi.org/10.1049/cvi2.12266

In response to the challenges of Multi-Object Tracking (MOT) in sports scenes, such as severe occlusions, similar appearances, drastic pose changes, and complex motion patterns, a deep-learning framework CTGMOT (CNN-Transformer-GNN-based MOT) specifically for multiple athlete tracking in sports videos that performs joint modelling of detection, appearance and motion features is proposed. Firstly, a detection network that combines Convolutional Neural Networks (CNN) and Transformers is constructed to extract both local and global features from images. The fusion of appearance and motion features is achieved through a design of parallel dual-branch decoders. Secondly, graph models are built using Graph Neural Networks (GNN) to accurately capture the spatio-temporal correlations between object and trajectory features from inter-frame and intra-frame associations. Experimental results on the public sports tracking dataset SportsMOT show that the proposed framework outperforms other state-of-the-art methods for MOT in complex sport scenes. In addition, the proposed framework shows excellent generality on benchmark datasets MOT17 and MOT20. The authors propose a deep-learning framework, CTGMOT, for multi-object tracking (MOT) in complex team sports videos. The backbone network of the framework combines CNN and Transformers to extract local and global features, and uses parallel decoders to fuse appearance and motion features. To accurately capture spatial-temporal correlations, the framework adopts GNN and an attention mechanism to fuse the spatial tracking features of objects within frames as well as the temporal tracking features across different frames, which better distinguishes fast-moving and occluded targets and improves the performance of online MOT.image