Telomeres, telomerase and tankyrase (TNKS) have an extremely important and special association with human cell aging and cancer. Telomerase activity is abnormally high in cancer cells and is accompanied by the overexpression of tankyrase 1 (TNKS1). TNKS1 is a positive regulator of telomerase activation and telomere extension in the human body, indicating that TNKS1 may be a possible therapeutic target for cancer. XAV939 is a small-molecule inhibitor of TNKS1. The objective of the present study was to investigate the apoptotic effect of XAV939 on the neuroblastoma (NB) SH-SY5Y cell line, as well as the change in telomere length and telomerase activity and elucidate the mechanism from this perspective. In the present study, we initially treated SH-SY5Y cells with XAV939 and RNA interference (RNAi)-TNKS1, and subsequently chose the optimal sequence for RNAi-TNKS1. We then measured the telomere length using quantitative real-time polymerase chain reaction (qPCR) assay, detected the telomerase activity using the ELISA kit, observed apoptotic morphology by transmission electron microscopy, and detected the percentages of apoptotic cells using flow cytometry and Hoechst 33342 staining. We also determined the invasive ability by a cell invasion assay. The results showed that short hairpin RNA-2 (shRNA-2) was the optimal sequence for RNAi-TNKS1. Treatment with both XAV939 and RNAi-TNKS1 shortened the telomere length, promoted apoptosis and reduced the invasive ability of the SH-SY5Y cells, yet had no effect on telomerase activity. XAV939 promoted apoptosis and reduced the invasiveness of SH-SY5Y cells dependent on telomere shortening, and further research should be conducted to clarify the exact mechanisms. This research may contribute to the cure of malignant NB using multi-targeted therapy with small-molecule agents.