Many animals and plants show left-right (LR) asymmetry. The LR asymmetry of mirror-image flowers has clear functional significance, with the reciprocal placement of male and female organs in left- versus right-handed flowers promoting cross-pollination. Here, we study how handedness of mirror-image flowers is determined and elaborated during development in the South African geophyte Cyanella alba. Inflorescences of C. alba produce flowers with a largely consistent handedness. However, this handedness has no simple genetic basis and individual plants can switch their predominant handedness between years. Rather, it is the direction of the phyllotactic spiral that predicts floral handedness. Style deflection is driven by increased cell expansion in the adaxial carpel facing the next oldest flower compared to the other adaxial carpel. The more expanding carpel shows transcriptional signatures of increased auxin signaling and auxin application can reverse the orientation of style deflection. We propose that a recently described inherent LR auxin asymmetry in the initiating organs of spiral phyllotaxis determines handedness in C. alba, creating a stable yet non-genetic floral polymorphism. This mechanism links chirality across different levels of plant development and exploits a developmental constraint in a core patterning process to produce morphological variation of ecological relevance.