Tongue diagnosis is an important practice in traditional Chinese medicine (TCM) for diagnosing diseases before determining proper means of treatments. Traditionally, it depends solely on personal knowledge and experience of the practitioner, thereby being criticized as lacking of objectivity. Currently, no research regarding intra- and inter-agreements of automatic tongue diagnosis system (ATDS) and TCM doctors has been conducted. In this study, the ATDS is developed to extract a variety of tongue features and provide practitioners with objective information to assist diagnoses. To evaluate the ATDS clinical stability, 2 sets of tongue images taken 1 hour apart from 20 patients with possible variations in lighting and extruding tongue, are employed to investigate intra-agreement of the ATDS, intra-agreement of the TCM doctors, and the inter-agreement between the ATDS and TCM doctors. The ATDS is shown to be more consistent with significantly higher intra-agreement than the TCM doctors (kappa value: 0.93 ± 0.06 versus 0.64 ± 0.13) with P < 0.001 (Student's t-test). Inter-agreements between the ATDS and TCM doctors, as well as among the TCM doctors are both moderate. The high agreement of the ATDS can provide objective and reliable tongue features to facilitate doctor in making effective observation and diagnosis of specific diseases.