Although pollen structure and morphology evolved toward the optimization of stability and fertilization efficiency, its performance is affected by harsh environmental conditions, e.g., heat, cold, drought, pollutants, and other stressors. These phenomena are expected to increase in the coming years in relation to predicted environmental scenarios, contributing to a rapid increase in the interest of the scientific community in understanding the molecular and physiological responses implemented by male gametophyte to accomplish reproduction. Here, after a brief introduction summarizing the main events underlying pollen physiology with a focus on polyamine involvement in its development and germination, we review the main effects that environmental stresses can cause on pollen. We report the most relevant evidence in the literature underlying morphological, cytoskeletal, metabolic and signaling alterations involved in stress perception and response, focusing on the final stage of pollen life, i.e., from when it hydrates, to pollen tube growth and sperm cell transport, with these being the most sensitive to environmental changes. Finally, we hypothesize the molecular mechanisms through which polyamines, well-known molecules involved in plant development, stress response and adaptation, can exert a protective action against environmental stresses in pollen by decoding the essential steps and the intersection between polyamines and pollen tube growth mechanisms.