Based on the aqueous sol-gel method, three types of nano-FeO particles with different microstructures were synthesized with FeSO solution as the precursor. The structural characteristics of these FeO particles and their effects on the high-temperature thermal decomposition of ammonium perchlorate (AP) and the combustion performance of AP/aluminum (Al) composite fuels were analyzed with scanning electron microscopy (SEM), energy dispersive spectroscopy (EDS), X-ray diffraction (XRD), surface area analysis, differential scanning calorimetry (DSC), and high-speed imaging. The results indicate that at a pH of 8, the synthesized nanoparticles comprise bulk and fibrous particles with an average particle size of approximately 70 nm, exhibiting some degrees of agglomeration. As the pH increases, the particle size shows a growing trend, with a gradual reduction in the quantity of fibrous particles. At a pH of 12, the morphology of the synthesized nanoparticles is predominantly bulk, characterized by an average particle size of 80 nm and no significant agglomeration; the bulk particles are identified as FeO, while the fibrous particles are FeOOH. With increasing pH, the specific surface area of the samples exhibits a decreasing trend, with no significant differences in specific surface area observed between the samples prepared at pH 10 and pH 12. The addition of three different sets of FeO nanoparticles led to a noticeable reduction in the high-temperature decomposition peak temperatures of AP, which decreased to 363.82 °C, 389.46 °C, and 404.54 °C, corresponding to reductions of 89.2 °C, 63.56 °C, and 48.48 °C, respectively; no significant differences were observed in the low-temperature decomposition peak temperatures. The heat release of the samples was measured at 635.56 J g, 1040.66 J g, and 985.34 J g, showing improvements of 60.08 J g, 465.18 J g, and 409.86 J g over pure AP. The maximum weight loss rates for the three samples were recorded at 363.91 °C, 388.76 °C, and 401.13 °C, corresponding to 69.1% min, 18% min, and 14% min. The combustion process of the composite fuels was divided into five stages: ignition, flaming, steady burning, decay, and extinguishment. The ignition delay time of composite fuels containing the nano-FeO catalyst decreased as the content of nano-FeO increased from 0.5 wt% to 1 wt% and 2 wt%, decreasing to 130 ms, 114 ms, and 95 ms, respectively, compared to reductions of 2 ms, 18 ms, and 37 ms for the composite fuel without nano-FeO. With the increased addition of nano-FeO particles, both the flame height and width during the steady burning stage increased significantly, resulting in a substantial enhancement of combustion duration. When the addition of nano-FeO reached 1 wt%, further increases in the particle content resulted in a stabilization of the flame area with no significant increases. Employing nano-FeO as a catalyst effectively enhances the combustion efficiency of composite fuels using AP as an oxidizer, suggesting good application prospects for nano-FeO particles as catalysts in solid rocket propellants.