Most accidents are a result of distractions while driving and road user's safety is a global concern. The proposed approach integrates advanced deep learning for driver distraction detection with real-time road object recognition to jointly address this problem. The behaviour of a driver is categorized into physical and visual distraction and cognitive distraction using Convolutional Neural Networks (CNN's) and transfer learning in order to achieve greater accuracy while also consuming lesser computational resources. The YOLO (You Only Look Once) detects vehicles, pedestrians, lane markers, and traffic signals, in real-time. This system has (1) distraction detection and (2) output (or scenario) recognition, which combine other systems and codes to evaluate driving scenarios. A decision-making module evaluates the combined data to assess danger levels and prompt either timely warnings or corrective actions. Our integrated solution will enable a fully-context capable Advanced Driver Assistance System (ADAS) to warn drivers of distractions and hazards, and increases overall situational awareness and reduces accidents. The methodology is supplemented by annotated pictures and videos of driver behavior and road situations in the rain, fog, and low-light scenarios. System reliability under a range of driving scenarios is achieved through data augmentation, model optimization, and transfer learning. State Farm Distracted Driver Dataset, KITTI and MS COCO benchmarks demonstrated better accuracy and efficiency. Integrating existing systems that monitor drivers with systems that are aware of the road would create a multi-target, comprehensive solution that makes both driving safer and helps build upon existing ADAS technology for the better. A real-time and scalable road safety system is established through the integration of CNNs and YOLO deep learning advances. The system's practicality was further validated through real-time embedded deployment on an NVIDIA Jetson Xavier NX platform, achieving 25 frames per second (FPS) with reduced latency and memory footprint, demonstrating feasibility for resource-constrained Advanced Driver Assistance Systems (ADAS). This paper presents a domain-specific driver monitoring module and a knowledge-based road hazard recognition model that better connect autonomous driving to the human side.