There is a growing consensus that math anxiety highly correlates with trait anxiety and that the emotional component elicited by math anxiety affects math performance. Yet few studies have examined the impact of "specific math anxiety" (high math anxiety and low other kinds of anxiety) on math performance and the underlying physiological and affective mechanism. The present study examines the mediation effect of heart rate variability-an affective measurement indexed by respiratory sinus arrhythmia (RSA)-in the relationship between specific math anxiety and arithmetic speed. A total of 386 junior high school students completed a self-reported questionnaire to measure their anxiety level. Among this sample, 29 individuals with specific math anxiety (high math anxiety and low reading and trait anxiety), 29 with specific reading anxiety (high reading anxiety and low math and trait anxiety), 24 with specific trait anxiety (high trait anxiety and low math and reading anxiety), and 22 controls (low math, trait and reading anxiety) were selected to participate in an arithmetic task and a reading task while RSA was recorded when they performed the tasks. Results revealed that individuals with specific math anxiety showed lower RSA and longer reaction time than the other three groups in the arithmetic task. Regression and mediation analyses further revealed that RSA mediated the relation between specific math anxiety and arithmetic speed. The present study provides the first account of evidence for the affective hypothesis of specific math anxiety and suggests that affective responses may be an important mechanism underlying the detrimental effect of specific math anxiety on math performance.