The concept of sustainable development has led to a growing research interest in bio-based flame-retardant polyurethane foams. These foams offer environmentally friendly, sustainable, and flame-retardant raw materials for the construction, automotive, and furniture industries. The 15 wt% soybean oil-based polyol rigid polyurethane foams (RPUF-S3A20B system) were modified with 20 wt% ammonium polyphosphate (APP) and homemade magnesium borate hydroxide (MgBO(OH)). Thermogravimetric analysis, pyrolysis kinetic analysis, limiting oxygen index (LOI) test, cone calorimetry (CONE), morphological analysis, and smoke density (Ds) were employed to investigate the impact of MgBO(OH) on the thermal stability and flame-retardant properties of the RPUF-S3A20B system. The results indicated that RPUF-S3A20B12.5 with 12.5 wt% MgBO(OH) had better thermal stability and higher activation energy. In addition, its LOI was increased by 3.1% compared to RPUF-S3A20 without MgBO(OH). The peak heat release rate (PHRR) and total heat release rate (THR) of RPUF-S3A20B12.5 at a radiant flux of 25 kW/m were reduced by 45.8% and 35.0% compared with RPUF-S3A20. RPUF-S3A20B12.5 demonstrated the lowest smoke density (17.4 and 17.5) and highest light transmission (73.9% and 73.7%) in both flameless and flame conditions, indicating superior flame-retardant and smoke-suppression properties. These findings offered valuable insights for further research on synergistic flame-retardant systems in bio-based polyurethane foams.