Cardiovascular diseases are the leading cause of death around the globe. In recent years, a crucial role of the immune system has been acknowledged in cardiac disease progression, opening the door for immunomodulatory therapies. To this ongoing change of paradigm, positron emission tomography (PET) imaging of the immune system has become a remarkable tool to reveal immune cell trafficking and monitor disease progression and treatment response. Currently, PET imaging of the immune system in cardiovascular disease mainly focuses on the innate immune system such as macrophages, while the immune cells of the adaptive immune system including B and T cells are less studied. This can be ascribed to the lack of radiotracers specifically binding to B and T cell biomarkers compatible with PET imaging within the cardiovascular system. In this review, we summarize current knowledge about the role of the adaptive immune system (e.g., B and T cells) in major cardiovascular diseases and introduce key biomarkers for specific targeting of these immune cells and their subpopulations. Finally, we present available radiotracers for these biomarkers and propose a pathway for developing probes or optimizing those already used in other fields (e.g., oncology) to make them compatible with the cardiovascular system.