This study examines the discrimination of Mandarin vowels and tones by native English speakers with varying amounts of Mandarin experience, aiming to investigate the relative difficulty of these two types of sounds for English speakers at different learning stages, and the source of their difficulty. Seventeen advanced learners of Mandarin (Ex group), eighteen beginning learners (InEx group), and eighteen English speakers naïve to Mandarin (Naïve group) participated in an AXB discrimination task. The stimuli were two Mandarin vowel contrasts, /li-ly/ and /lu-ly/, and two tonal contrasts, T1-T4 and T2-T3. The predicted difficulty for each contrast was hypothesized based on the assimilation of these sounds to English reported in previous work. The results showed that the Naïve group was more accurate with vowel contrasts than with tones, suggesting that non-tonal language speakers without any Mandarin training are less sensitive to tonal distinction than to vowels. The two learner groups, on the other hand, were highly accurate with all contrasts except for the T2-T3 pair, and achieved significantly higher accuracy than the Naïve group on /li-ly/ and T1-T4. This lends support to the view that experience in Mandarin improves English speakers' sensitivity to tonal distinction, helping them discriminate some tones as accurately as vowels. However, all three groups achieved low accuracy in discriminating T2 and T3, suggesting that this contrast may be inherently difficult and resistant to improvement. This study shows that various factors in addition to the native language experience may affect the perception of non-native vowels and tones.