Skin cancer is among the 10 most common cancers. Recent research revealed the superiority of artificial intelligence (AI) over dermatologists to diagnose skin cancer from predesignated and cropped images. However, there remain several uncertainties for AI in diagnosing skin cancers, including lack of testing for consistency, lack of pathological proof or ambiguous comparisons. Hence, to develop a reliable, feasible and user-friendly platform to facilitate the automatic diagnostic algorithm is important. The aim of this study was to build a light-weight skin cancer classification model based on deep learning methods for aiding first-line medical care. The developed model can be deployed on cloud platforms as well as mobile devices for remote diagnostic applications. We reviewed the medical records and clinical images of patients who received a histological diagnosis of basal cell carcinoma, squamous cell carcinoma, melanoma, seborrheic keratosis and melanocytic nevus in 2006-2017 in the Department of Dermatology in Kaohsiung Chang Gung Memorial Hospital (KCGMH). We used the deep learning models to identify skin cancers and benign skin tumors in the manner of binary classification and multi-class classification in the KCGMH and HAM10000 datasets to construct a skin cancer classification model. The accuracy reached 89.5% for the binary classifications (benign vs malignant) in the KCGMH dataset; the accuracy was 85.8% in the HAM10000 dataset in seven-class classification and 72.1% in the KCGMH dataset in five-class classification. Our results demonstrate that our skin cancer classification model based on deep learning methods is a highly promising aid for the clinical diagnosis and early identification of skin cancers and benign tumors.