Previous research suggests that a history of early life stress (ELS) impacts working memory (WM) in adulthood. Despite the widespread use of WM paradigms, few studies have evaluated whether ELS exposure, in the absence of psychiatric illness, also impacts WM-associated brain activity in ways that might improve sensitivity to these ELS effects or provide insights into the mechanisms of these effects. This study evaluated whether ELS affects WM behavioral performance and task-associated activity by acquiring 3T functional images from 27 medication-free healthy adults (14 with ELS) during an N-back WM task that included 0- and 2-back components. Whole brain voxel-wise analysis was performed to evaluate WM activation, followed by region of interest analyses to evaluate relationships between activation and clinical variables. ELS was associated with poorer accuracy during the 2-back (79% ± 19 vs. 92% ± 9, p = 0.049); accuracy and response time otherwise did not differ between groups. During the 0-back, ELS participants demonstrated increased activation in the superior temporal gyrus/insula, left inferior parietal lobule (IPL) (both corrected p < 0.001), and middle temporal and parahippocampal gyrus (MTG/PHG)(corrected p < 0.010). During the 2-back, ELS was associated with greater activation in the IPL, MTG/PHG and inferior frontal gyrus (corrected p < 0.001), with a trend towards precuneus activation (p = 0.080). These findings support previous research showing that ELS is associated with impaired neurobehavioral performance and changes in brain activation, suggesting recruitment of additional cognitive resources during WM in ELS. Based on these findings, ELS screening in future WM imaging studies appears warranted.