Indoleamine 2,3-dioxygenase (IDO) has been reported to be involved in esophageal squamous cell cancer (ESCC) progression by promoting immune escape. Previous studies have revealed bridging integrator-1 (Bin1) can inhibit cancer cell growth by suppressing expression of IDO, thus we investigated the correlation between the expression of Bin1 and IDO and their prognostic significances for ESCC patients. Specimens were collected from 196 ESCC patients and detected with flow cytometry, reverse transcription-polymerase chain reaction and immunohistochemistry. We found that in tumor microenvironment (TME) and tumor draining lymph node (TDLN), the proportions of CD3(+) CD4(+) T cell, CD3(+) CD8(+) T cell and CD3(-) CD16(+) CD56(+) NK cell were lower while the proportions of CD3(-) CD19(+) B cell and CD4(+) CD25(+) Treg were higher in specimens with high IDO expression when compared to the specimens with low IDO expression (p < 0.01). In addition, IDO expression was negatively correlated with Bin1 expression at gene and protein level in TME and TDLN. Both the expression of Bin1 and IDO were associated with some clinicopathological parameters including differentiation grade, TNM stage, invasion range, lymph node metastasis (p < 0.05). Moreover, multivariate survival analysis suggested that, along with some other parameters, low expression of Bin1 and high expression of IDO might be independent prognostic factor for ESCC patients. Our results demonstrate that low expression of Bin1, along with high expression of IDO, are predictor for poor prognosis in ESCC and thereby could be used to establish new therapeutic strategies.