In déjà vu, a phenomenological impression of familiarity for the current visual environment is experienced with a sense that it should in fact not feel familiar. The fleeting nature of this phenomenon in daily life, and the difficulty in developing experimental paradigms to elicit it, has hindered progress in understanding déjà vu. Some neurological patients with temporal-lobe epilepsy (TLE) consistently experience déjà vu at the onset of their seizures. An investigation of such patients offers a unique opportunity to shed light on its possible underlying mechanisms. In the present study, we sought to determine whether unilateral TLE patients with déjà vu (TLE+) show a unique pattern of interictal memory deficits that selectively affect familiarity assessment. In Experiment 1, we employed a Remember-Know paradigm for categorized visual scenes and found evidence for impairments that were limited to familiarity-based responses. In Experiment 2, we administered an exclusion task for highly similar categorized visual scenes that placed both recognition processes in opposition. TLE+ patients again displayed recognition impairments, and these impairments spared their ability to engage recollective processes so as to counteract familiarity. The selective deficits we observed in TLE+ patients contrasted with the broader pattern of recognition-memory impairments that was present in a control group of unilateral patients without déjà vu (TLE-). MRI volumetry revealed that ipsilateral medial temporal structures were less broadly affected in TLE+ than in TLE- patients, with a trend for more focal volume reductions in the rhinal cortices of the TLE+ group. The current findings establish a first empirical link between déjà vu in TLE and processes of familiarity assessment, as defined and measured in current cognitive models. They also reveal a pattern of selectivity in recognition impairments that is rarely observed and, thus, of significant theoretical interest to the memory literature at large.