Default mode network (DMN) activity, measured with fMRI, typically increases during internally directed thought, and decreases during tasks that demand externally focused attention. However,Crittenden et al. (2015)andSmith et al. (2018)reported increased DMN activity during demanding external task switches between different cognitive domains, compared to within-domain switches and task repeats. This finding is hard to reconcile with many dominant views of DMN function. Here, we aimed to replicate this DMN task-switch effect in a similar paradigm and test whether it reflects increased representation of broader context, specifically of a scene presented behind the focal task. In the Core DMN, we found significant activity for all task switches, compared to task repeats, and stronger activity for switches between rest and task. Although the content of the background scene was attended, recalled, and neurally decodable, there was no evidence that this differed by switch type. Therefore, external task switches activated DMN without enhanced processing of the surrounding background. Surprisingly, DMN activity at within-domain switches was no less than at between-domain switches. We suggest that modulation of DMN activity by task switches reflects a shift in the current cognitive model and depends on the overall complexity of that model.