This paper introduces SeRankDet, a deep learning model for infrared small target detection. SeRankDet leverages our novel Selective Rank-Aware Attention module for preserving salient responses, Large Selective Feature Fusion for dynamic feature integration, and Dilated Difference Convolution for improved target-background separation. Despite its lightweight design, SeRankDet delivers state-of-the-art performance.