Heavy metal pollution in cultivated land poses a serious threat to environmental health and farmers' livelihoods. As the direct user of cultivated land, understanding farmers' adaptive behavior to heavy metal pollution, and its influencing factors, can provide insight and information relevant for decision-making, so as to better manage the hazards and risks of heavy metal pollution. We proposed a conceptual framework of "farmers' characteristics-perceptions-adaptive behaviors". Factor analysis and mediation effect analysis were used to explore the influence of characteristics and perceptions on adaptive behaviors. The data of 278 farmers in a typical mining area in Daye, China, show that local farmers perceive the hazards of heavy metal pollution, but their adaptive behaviors are hindered to a certain extent. The results of the mediation effect analysis show that perceptions of health impact, self-efficacy, and adaptive cost play a partial mediating role in the impact of characteristics on adaptive behaviors. In addition, the influence of the "factor of dependence on farmland" and the "factor of obstacles to action" on adaptive behavior have no significant relationship with perception levels. By comparing the influencing factors, we found that although farmers' perceptions have mediating effects between characteristics and adaptive behaviors, characteristics still play a decisive role in adaptive behaviors.