IDSelect: An RL-based Cost-Aware Selection Agent
for Video-based Person Recognition

Yuyang Ji¹, Yixuan Shen¹, Kien Nguyen³, Lifeng Zhou², Feng Liu¹⁺

+ Corresponding author

¹Dept. of CS, Drexel University ²Dept. of ECE, Drexel University ³School of EE&R, Queensland University of Technology

Top: Traditional multi-modal person recognition methods (e.g., QME) use fixed model combinations for all inputs, while our IDSelect employs an RL-based cost-aware agent to adaptively select complementary models from diverse pools based on input characteristics. Bottom: Accuracy vs. computational cost on CCVID dataset. Our method achieves superior accuracy (95.9%, +1.8%) with 92.4% fewer FLOPs than QME.

Abstract

Video-based person recognition achieves robust identification by integrating face, body, and gait. However, current systems waste computational resources by processing all modalities with fixed heavyweight ensembles regardless of input complexity. To address these limitations, we propose IDSelect, a reinforcement learning-based cost-aware selector that chooses one pre-trained model per modality per-sequence to optimize the accuracy-efficiency trade-off. Our key insight is that an input-conditioned selector can discover complementary model choices that surpass fixed ensembles while using substantially fewer resources. IDSelect trains a lightweight agent end-to-end using actor-critic reinforcement learning with budget-aware optimization. The reward balances recognition accuracy with computational cost, while entropy regularization prevents premature convergence. At inference, the policy selects the most probable model per modality and fuses modality-specific similarities for the final score. Extensive experiments on challenging video-based datasets demonstrate IDSelect's superior efficiency: on CCVID, it achieves 95.9% Rank-1 accuracy with 92.4% less computation than strong baselines while improving accuracy by 1.8%; on MEVID, it reduces computation by 41.3% while maintaining competitive performance.

Key Findings

95.9%

Rank-1 Accuracy on CCVID

92.4%

Computation Reduction on CCVID

+1.8%

Accuracy Improvement over Baselines

41.3%

Computation Reduction on MEVID

Method Overview

IDSelect introduces an RL-based cost-aware agent that dynamically selects one pre-trained model per modality for each input, discovering complementary model combinations that outperform static ensembles while using significantly fewer computational resources.

RL-based Cost-Aware Selection: IDSelect employs an RL-based cost-aware agent that discovers complementary model combinations across face, gait, and body modalities for efficient whole-body recognition.
Actor-Critic Training with Budget Control: A lightweight selection agent is trained end-to-end via an actor-critic reinforcement learning framework with a Lagrangian budget controller, enabling stable and budget-aware model selection.
Superior Accuracy-Cost Trade-offs: Extensive experiments on video-based whole-body datasets demonstrate that IDSelect discovers superior model combinations, achieving better accuracy-cost trade-offs than fixed and quality-guided fusion baselines.

Results

Selection frequency of each model across face, gait, and body modalities on the CCVID dataset. IDSelect learns to select complementary models across modalities, demonstrating input-adaptive selection behavior.

BibTeX

@article{ji2026idselect,
  title={IDSelect: An RL-based Cost-Aware Selection Agent for Video-based Person Recognition},
  author={Ji, Yuyang and Shen, Yixuan and Nguyen, Kien and Zhou, Lifeng and Liu, Feng},
  journal={arXiv preprint},
  year={2026}
}