Prostate cancer (PC) is a notoriously immune-cold tumor in that it often lacks substantial infiltration by antitumor immune cells, and in advanced diseases such as neuroendocrine PC, it could be devoid of immune cells. A majority of PC patients thus have, unfortunately, been unable to benefit from recent advances in immunotherapies. What causes this immunosuppressive microenvironment around PC? In this review, we discuss various genetic and epigenetic regulators intrinsic to prostate tumor cells that could have profound effects on the tumor microenvironment, thus contributing to this immune-cold status. It will be essential to target the cancer cells themselves in order to change the tumor microenvironment to harness existing and developing immunotherapies that had great success in other tumors.