Sensory loss leads to widespread adaptation of neural circuits to mediate cross-modal plasticity, which allows the organism to better utilize the remaining senses to guide behavior. While cross-modal interactions are often thought to engage multisensory areas, cross-modal plasticity is often prominently observed at the level of the primary sensory cortices. One dramatic example is from functional imaging studies in humans where cross-modal recruitment of the deprived primary sensory cortex has been observed during the processing of the spared senses. In addition, loss of a sensory modality can lead to enhancement and refinement of the spared senses, some of which have been attributed to compensatory plasticity of the spared sensory cortices. Cross-modal plasticity is not restricted to early sensory loss but is also observed in adults, which suggests that it engages or enables plasticity mechanisms available in the adult cortical circuit. Because adult cross-modal plasticity is observed without gross anatomical connectivity changes, it is thought to occur mainly through functional plasticity of pre-existing circuits. The underlying cellular and molecular mechanisms involve activity-dependent homeostatic and Hebbian mechanisms. A particularly attractive mechanism is the sliding threshold metaplasticity model because it innately allows neurons to dynamically optimize their feature selectivity. In this mini review, I will summarize the cellular and molecular mechanisms that mediate cross-modal plasticity in the adult primary sensory cortices and evaluate the metaplasticity model as an effective framework to understand the underlying mechanisms.