This work explores the possibility of increasing the density of negatively charged nitrogen-vacancy centers ([NV(-)]) in nanodiamonds using nitrogen-rich type Ib diamond powders as the starting material. The nanodiamonds (10-100 nm in diameter) were prepared by ball milling of microdiamonds, in which the density of neutral and atomically dispersed nitrogen atoms ([N(0)]) was measured by diffuse reflectance infrared Fourier transform spectroscopy. A systematic measurement of the fluorescence intensities and lifetimes of the crushed monocrystalline diamonds as a function of [N(0)] indicated that [NV(-)] increases nearly linearly with [N(0)] at 100-200 ppm. The trend, however, failed to continue for nanodiamonds with higher [N(0)] (up to 390 ppm) but poorer crystallinity. We attribute the result to a combined effect of fluorescence quenching as well as the lower conversion efficiency of vacancies to NV(-) due to the presence of more impurities and defects in these as-grown diamond crystallites. The principles and practice of fabricating brighter and smaller fluorescent nanodiamonds are discussed.