We present a first-principles method for the calculation of the temperature-dependent relaxation of symmetry-breaking atomic driving forces in photoexcited systems. We calculate the phonon-assisted decay of the photoexcited force on the low-symmetry E_{g} mode following absorption of an ultrafast pulse in Bi, Sb, and As. The force decay lifetimes for Bi and Sb are of the order of 10 fs and in agreement with recent experiments, demonstrating that electron-phonon scattering is the primary mechanism relaxing the symmetry-breaking forces. Calculations for a range of absorbed photon energies suggest that larger amplitude, symmetry-breaking atomic motion may be induced by choosing a pump photon energy which maximizes the product of the initial E_{g} force and its lifetime. The high-symmetry A_{1g} force undergoes a partial decay to a nonzero constant on similar timescales, which has not yet been measured in experiments. The average imaginary part of the electron self-energy over the photoexcited carrier distribution provides a crude indication of the decay rate of symmetry-breaking forces.