Convolutional neural network (CNN) models have been extensively applied to skin lesions segmentation due to their information discrimination capabilities. However, CNNs' struggle to capture the connection between long-range contexts when extracting deep semantic features from lesion images, resulting in a semantic gap that causes segmentation distortion in skin lesions. Therefore, detecting the presence of differential structures such as pigment networks, globules, streaks, negative networks, and milia-like cysts becomes difficult. To resolve these issues, we have proposed an approach based on semantic-based segmentation (Dermo-Seg) to detect differential structures of lesions using a UNet model with a transfer-learning-based ResNet-50 architecture and a hybrid loss function. The Dermo-Seg model uses ResNet-50 backbone architecture as an encoder in the UNet model. We have applied a combination of focal Tversky loss and IOU loss functions to handle the dataset's highly imbalanced class ratio. The obtained results prove that the intended model performs well compared to the existing models. The dataset was acquired from various sources, such as ISIC18, ISBI17, and HAM10000, to evaluate the Dermo-Seg model. We have dealt with the data imbalance present within each class at the pixel level using our hybrid loss function. The proposed model achieves a mean IOU score of 0.53 for streaks, 0.67 for pigment networks, 0.66 for globules, 0.58 for negative networks, and 0.53 for milia-like-cysts. Overall, the Dermo-Seg model is efficient in detecting different skin lesion structures and achieved 96.4% on the IOU index. Our Dermo-Seg system improves the IOU index compared to the most recent network.