Age-related inflammation or 'inflammaging' increases disease burden and controls lifespan. Adipose tissue macrophages (ATMs) are critical regulators of inflammaging; however, the mechanisms involved are not well understood in part because the molecular identities of niche-specific ATMs are unknown. Using intravascular labeling to exclude circulating myeloid cells followed by single-cell sequencing with orthogonal validation via multiparametric flow cytometry, we define sex-specific changes and diverse populations of resident ATMs through lifespan in mice. Aging led to depletion of vessel-associated macrophages, expansion of lipid-associated macrophages and emergence of a unique subset of CD38 age-associated macrophages in visceral adipose tissue with inflammatory phenotype. Notably, CD169CD11c ATMs are enriched in a subpopulation of nerve-associated macrophages (NAMs) that declines with age. Depletion of CD169 NAMs in aged mice increases inflammaging and impairs lipolysis suggesting catecholamine resistance in visceral adipose tissue. Our findings reveal NAMs are a specialized ATM subset that control adipose homeostasis and link inflammation to tissue dysfunction during aging.