In understanding how humans selectively attend, common frameworks often focus on how attention is allocated relative to an idealized allocation based on properties of the task. However, these perspectives often ignore different types of constraints that could help explain why attention was allocated in a particular way. For example, many computational models of learning are well equipped to explain how attention should ideally be allocated to minimize errors within the task, but these models often assume all features are perfectly encoded or that the only learning goal is to maximize accuracy. In this article, we argue for a more comprehensive view by using computational modeling to understand the complex interactions that occur between selective attention and memory. Our central thesis is that, although selective attention directs attention to relevant dimensions, relevance can only be established through memories of previous experiences. Hence, attention is initially used to encode features and create memories, but thereafter, attention operates selectively based on what is kept in memory. Through this lens, deviations from ideal performance can still be viewed as goal directed selective attention, but the orientation of attention is subject to the constraints of the individual learner.