Diffusion MRI affords valuable insights into white matter microstructures, but suffers from low signal-to-noise ratio (SNR), especially at high diffusion weighting (i.e., b-value). To avoid time-intensive repeated acquisition, post-processing algorithms are often used to reduce noise. Among existing methods, non-local means (NLM) has been shown to be particularly effective. However, most NLM algorithms for diffusion MRI focus on patch matching in the spatial domain (i.e., x-space) and disregard the fact that the data live in a combined 6D space covering both spatial domain and diffusion wavevector domain (i.e., q-space). This drawback leads to inaccurate patch matching in curved white matter structures and hence the inability to effectively use recurrent information for noise reduction. The goal of this paper is to overcome this limitation by extending NLM to the joint x-q space. Specifically, we define for each point in the x-q space a spherical patch from which we extract rotation-invariant features for patch matching. The ability to perform patch matching across q-samples allows patches from differentially orientated structures to be used for effective noise removal. Extensive experiments on synthetic, repeated-acquisition, and HCP data demonstrate that our method outperforms state-of-the-art methods, both qualitatively and quantitatively.