The International Health Regulations-State Party Annual Reporting (IHR-SPAR) index and the Global Health Security Index (GHSI) have been developed to aid in strengthening national capacities for pandemic preparedness. We examined the relationship between country-level rankings on these two indices, along with two additional indices (the Universal Health Coverage Service Coverage Index and World Bank Worldwide Governance Indicator (n = 195)) and compared them to the country-level reported coronavirus disease (COVID-19) cases and deaths (Johns Hopkins University COVID-19 Dashboard) through 17 June 2020. Ordinary least squares regression models were used to compare weekly reported COVID-19 cases and death rates per million in the first 12 weeks of the pandemic between countries classified as low, middle and high ranking on each index while controlling for country socio-demographic information. Countries with higher GHSI and IHR-SPAR index scores experienced fewer reported COVID-19 cases and deaths but only for the first 8 weeks after the country's first case. For the GHSI, this association was further limited to countries with populations below 69.4 million. For both the GHSI and IHR-SPAR, countries with a higher sub-index score in human resources for pandemic preparedness reported fewer COVID-19 cases and deaths in the first 8 weeks after the country's first reported case. The Universal Health Coverage Service Coverage Index and Worldwide Governance Indicator country-level rankings were not associated with COVID-19 outcomes. The associations between GHSI and IHR-SPAR scores and COVID-19 outcomes observed in this study demonstrate that these two indices, although imperfect, may have value, especially in countries with a population under 69.4 million people for the GHSI. Preparedness indices may have value; however, they should continue to be evaluated as policy makers seek to better prepare for future global public health crises.