Worldwide, cancer is one of the leading causes of death in humans. Interobserver variability and specialized experience are key factors in diagnosing gastrointestinal tract (GIT) abnormalities using endoscopic procedures. Due to this diversity, small lesions may go unnoticed, leading to a delay in early diagnosis. Therefore, it is essential to design a computer-aided diagnosis (CAD) system for the detection and classification of GIT diseases at the early stages. This paper proposes a CAD system that combines the feature fusion of modified deep learning models with optimal feature selection. Three publicly available datasets, including Kvasir V1, Kvasir V2, and Hyperkvasir, are utilized in the experimental process. In the proposed method, a contrast enhancement step is performed using the fusion of the top-bottom filtering technique. In the next step, two deep learning models (ResNet18 and ResNet50) are modified with a new layer called entropic field propagation (EFP). The pooling layers are replaced with EFP layers in both models, which are then trained on the selected datasets. In the testing process, trained models are employed, and features are extracted from the deeper layers, which are further refined using the Newton-Raphson Marine Predator Optimization (NRMPO) algorithm. The selected features from both models are finally fused using a novel mean threshold-based fusion approach and passed to machine learning classifiers. The proposed CAD system achieved accuracies of 99.0, 89.6, and 82.7% for Kvasir V1, Kvasir V2, and HyperKvasir, respectively. A detailed ablation study is also conducted for the middle steps that validate these reported accuracies. Conclusion: A comparison is performed with state-of-the-art (SOTA) techniques, showing that the proposed method achieves improved accuracy and precision rates.