Diffusion-weighted magnetic resonance imaging (dMRI) has found great utility for a wide range of neuroscientific and clinical applications. However, high-resolution dMRI, which is required for improved delineation of fine brain structures and connectomics, is hampered by its low signal-to-noise ratio (SNR). Since dMRI relies on the acquisition of multiple different diffusion weighted images of the same anatomy, it is well-suited for denoising methods that utilize correlations across the image series to improve the apparent SNR and the subsequent data analysis. In this work, we introduce and quantitatively evaluate a comprehensive framework, NOise Reduction with DIstribution Corrected (NORDIC) PCA method for processing dMRI. NORDIC uses low-rank modeling of g-factor-corrected complex dMRI reconstruction and non-asymptotic random matrix distributions to remove signal components which cannot be distinguished from thermal noise. The utility of the proposed framework for denoising dMRI is demonstrated on both simulations and experimental data obtained at 3 Tesla with different resolutions using human connectome project style acquisitions. The proposed framework leads to substantially enhanced quantitative performance for estimating diffusion tractography related measures and for resolving crossing fibers as compared to a conventional/state-of-the-art dMRI denoising method.