Obesity-associated inflammation is a systemic process that affects all metabolic organs. Prominent among these is adipose tissue, where cells of the innate and adaptive immune system are markedly changed in obesity, implicating these cells in a range of processes linking immune memory to metabolic regulation. Furthermore, weight loss and weight cycling have unexpected effects on adipose tissue immune populations. Here, we review the current literature on the roles of various immune cells in lean and obese adipose tissue. Within this context, we discuss pharmacological and nonpharmacological approaches to obesity treatment and their impact on systemic inflammation.