Long-term exposure to diabetes mellitus leads to the formation of diabetic retinopathy (DR), which can cause vision loss in working-age adults. Early stage diagnosis of DR is highly essential for preventing vision loss and preserving vision in people with diabetes. The motivation behind the severity grade classification of DR is to develop an automated system that can assist ophthalmologists and healthcare professionals in the diagnosis and management of DR. However, existing methods suffer from variability in image quality, similar structures of the normal and lesion regions, high dimensional features, variability in disease manifestations, small datasets, high training loss, model complexity, and overfitting, which leads to high misclassification errors in the severity grading system. Hence, there is a need to develop an automated system using improved deep learning techniques to provide a reliable and consistent grading of DR severity with high classification accuracy using fundus images. To solve these issues, we proposes a Deformable Ladder Bi attention U-shaped encoder-decoder network and Deep Adaptive Convolutional Neural Network (DLBUnet-DACNN) for accurate severity classification of DR. The DLBUnet performs lesion segmentation that can be divided into three parts: the encoder, the central processing module and the decoder. In the encoder part, deformable convolution is used instead of convolution to learn different shapes of the lesion by understanding the offset location. Afterwards, Ladder Atrous Spatial Pyramidal Pooling (LASPP) using variable dilation rates is introduced in the central processing module. LASPP enhance the tiny lesion features and variable dilation rates avoid gridding effects and can learn better global context information. Then the decoder part uses a bi-attention layer contains spatial and channel attention, which can learn contour and edges of the lesion accurately. Finally, the severity of DR is classified using a DACNN by extracting the discriminative features from the segmentation results. Experiments are conducted on the Messidor-2, Kaggle, and Messidor datasets. Our proposed method DLBUnet-DACNN achieves better results in terms of accuracy of 98.2, recall of 0.987, kappa coefficient of 0.993, precision of 0.98, F1-score of 0.981, Matthews Correlation Coefficient (MCC) of 0.93 and Classification Success Index (CSI) of 0.96 when compared to existing methods.