Mode coupling and resonance instabilities in quasi-two-dimensional dust clusters in complex plasmas.

Small quasi-two-dimensional dust clusters consisting of three to eleven particles are formed in an argon plasma under varying rf power. Their normal modes are investigated through their mode spectra obtained from tracking the particles' thermal motion. Detailed coupling patterns between their horizontal and vertical modes are detected for particle numbers up to 7 and discrete instabilities are found for dust clusters with particle number ⩾9, as predicted in previous theory on ion-flow induced mode coupling in small clusters. The instabilities are proven to be induced by resonance between coupled horizontal and vertical normal modes.