Recently, accurate diagnosis of thyroid nodules has played a critical role in precision medicine and healthcare system management. Due to complicated changes in ultrasound features of texture, and similar visual appearance of benign-malignant nodules, the identification of cancerous thyroid lesions from a given ultrasound image still faces challenges for even experienced radiologists. Learning-based computer-aided diagnosis (CAD) systems have accordingly attracted more and more attention recently. However, little research is committed to developing a deep learning-based CAD system that has greater conformity with radiologists' diagnostic decision-making. In this study, we devise a texture and shape focused dual-stream attention neural network, dubbed TS-DSANN. Specifically, in the texture focused stream, we utilize the ImageNet pre-trained ResNet34 to guide the network to recognize texture-related nodule attributes. Meanwhile, in the shape focused stream, in addition to using ResNet34 backbone, jointly learning from scratch with the contour obtained by contour detection module to enhance the extraction of shape features. Afterward, we employ a concatenation operation to aggregate the abovementioned two stream networks for capturing richer and more representative features. Finally, we further utilize an online class activation mapping mechanism to assist the dual-stream network in generating a localization heatmap to obtain more visualization attention to the nodule from the whole image, and supervise classifier's attention in decision-making. Experimental results conducted on the two-center thyroid nodule ultrasound datasets verify that our proposed method has improved the classification performance, superior to the state-of-the-art methods.