The high mobility of uncrewed aerial vehicles (UAVs) has led to their usage in various computer vision applications, notably in intelligent traffic surveillance, where it enhances productivity and simplifies the process. Yet, there are still several challenges that must be resolved to automate these systems. One significant challenge is the accurate extraction of vehicle foregrounds in complex traffic scenarios. As a result, this article proposes a novel vehicle detection and tracking system for autonomous vehicle surveillance, which employs Fuzzy C-mean clustering to segment the aerial images. After segmentation, we employed the YOLOv4 deep learning algorithm, which is efficient in detecting small-sized objects in vehicle detection. Furthermore, an ID assignment and recovery algorithm based on Speed-Up Robust Feature (SURF) is used for multi-vehicle tracking across image frames. Vehicles are determined by counting in each image to estimate the traffic density at different time intervals. Finally, these vehicles were tracked using DeepSORT, which combines the Kalman filter with deep learning to produce accurate results. Furthermore, to understand the traffic flow direction, the path trajectories of each tracked vehicle is projected. Our proposed model demonstrates a noteworthy vehicle detection and tracking rate during experimental validation, attaining precision scores of 0.82 and 0.80 over UAVDT and KIT-AIS datasets for vehicle detection. For vehicle tracking, the precision is 0.87 over the UAVDT dataset and 0.83 for the KIT-AIS dataset.