Neural stem cells (NSCs) in the mammalian brain decline rapidly with age, leading to impairment of hippocampal memory function in later life. However, the relationship between epigenetic remodeling and transcriptional regulation that compromises hippocampal NSC activity during the early stage of chronological aging remains unclear. Here, we performed single-cell RNA sequencing (scRNA-seq) and single-cell ATAC sequencing (scATAC-seq) on NSCs and newly generated neurons across different stages. Integrated data analysis revealed continuous alterations in the chromatin profile of hippocampal NSCs and their progeny from neonatal to mature adult stages, accompanied by consistent changes in transcriptional profiles. Further, decreased expression of Setd8, encoding the enzyme for histone H4 monomethylation at lysine 20 (H4K20me1), underlies age-related changes in mouse hippocampal NSCs. Notably, depletion of Setd8 elicits alterations in gene expression and epigenetic regulation that phenocopy age-related changes, and impairs NSC activity, leading to hippocampal memory deficits. Together, our study provides a global map of longitudinal chromatin and transcriptome changes during brain aging and identifies mechanistic insights into early-onset decline of NSC activity and hippocampal neurogenesis that precedes functional aging.