Magnetic particle imaging (MPI) is a novel non-invasive molecular imaging technology that images the distribution of superparamagnetic iron oxide nanoparticles (SPIONs). It is not affected by imaging depth, with high sensitivity, high resolution, and no radiation. The MPI reconstruction with high precision and high quality is of enormous practical importance, and many studies have been conducted to improve the reconstruction accuracy and quality. MPI reconstruction based on the system matrix (SM) is an important part of MPI reconstruction. In this review, the principle of MPI, current construction methods of SM and the theory of SM-based MPI are discussed. For SM-based approaches, MPI reconstruction mainly has the following problems: the reconstruction problem is an inverse and ill-posed problem, the complex background signals seriously affect the reconstruction results, the field of view cannot cover the entire object, and the available 3D datasets are of relatively large volume. In this review, we compared and grouped different studies on the above issues, including SM-based MPI reconstruction based on the state-of-the-art Tikhonov regularization, SM-based MPI reconstruction based on the improved methods, SM-based MPI reconstruction methods to subtract the background signal, SM-based MPI reconstruction approaches to expand the spatial coverage, and matrix transformations to accelerate SM-based MPI reconstruction. In addition, the current phantoms and performance indicators used for SM-based reconstruction are listed. Finally, certain research suggestions for MPI reconstruction are proposed, expecting that this review will provide a certain reference for researchers in MPI reconstruction and will promote the future applications of MPI in clinical medicine.