Falling poses a significant health risk to the elderly, often resulting in severe injuries if not promptly addressed. As the global population increases, the frequency of falls increases along with the associated financial burden. Hence, early detection is crucial for initiating timely medical interventions and minimizing physical, social, and economic harm. With the growing demand for safety monitoring of older adults, particularly those living alone, effective fall detection has become increasingly important for supporting independent living. In this study, we propose a novel deep learning architecture and an optimization algorithm for human fall direction recognition. Subsequently, we developed four novel residual block and self-attention mechanisms, named residual block-deep convolutional neural network (3-RBNet), 5-RBNet, 7-RBNet, and 9-RBNet self-attention models. The models were trained on enhanced images, and deep features were extracted from the self-attention layer. The 7-RBNet and 9-RBNet self-attention models demonstrated superior accuracy and precision rates, leading us to exclude the 3-RBNet self model from further analysis. To optimize feature selection and improve classification performance while reducing computational costs, we employed the tree seed algorithm on the self-attention features of 7-RBNet and 9-RBNet self-attention models. Experiments using the proposed method were performed on a human fall dataset collected from Soonchunhyang University, South Korea. The proposed method achieved maximum accuracies of 93.2% and 92.5%, respectively. Compared with recent techniques, our approach improved accuracy and precision.