To interpret complex sensory scenes, animals exploit statistical regularities to infer missing features and suppress redundant or ambiguous information. Cortical microcircuits might contribute to this cognitive goal by either completing or cancelling predictable activity, but it remains unknown whether, and how, a single circuit can implement these antagonistic computations. To address this central question, we used all-optical physiology to simulate sensory-evoked activity patterns in pyramidal cells (PCs) and somatostatin interneurons (SSTs) in the mouse's primary visual cortex (V1). In the absence of external visual input, photostimulation of orientation-tuned PC ensembles drove either completion or cancelation of input-matching representations, depending on the number of photostimulated cells. This dual computational capacity arose from the co-existence of 'like-to-like' excitatory interactions between PCs, and a newly discovered 'like-to-like' SST-PC connectivity motif, in which SSTs are preferentially recruited by, and in turn suppress, similarly tuned PCs. Finally, we show that photoactivation of tuned SST ensembles during visual processing improved the discriminability of their preferred visual input by suppressing ambiguous activity. Thus, these complementary feature-specific connectivity motifs allow different strategies of contextual modulation to optimize inference by either completion (through PC-PC interactions) or cancelation (via PC-SST-PC loops) of predictable activity, depending on the structure of the input and the network state.