Bioluminescence tomography (BLT) improves upon commonly-used 2D bioluminescence imaging by reconstructing 3D distributions of bioluminescence activity within biological tissue, allowing tumor localization and volume estimation-critical for cancer therapy development. Conventional model-based BLT is computationally challenging due to the ill-posed nature of the problem and data noise. We introduce a self-supervised hybrid neural network (SHyNN) that integrates the strengths of both conventional model-based methods and machine learning (ML) techniques to address these challenges. The network structure and converging path of SHyNN are designed to mitigate the effects of ill-posedness for achieving accurate and robust solutions. Through simulated and in vivo data on different disease sites, it is demonstrated to outperform the conventional reconstruction approach, particularly under high noise, in tumor localization, volume estimation, and multi-tumor differentiation, highlighting the potential towards quantitative BLT for cancer research.