Family caregiving is a public health issue because of caregivers' significant contribution to the health and social care systems, as well as the substantial impact that giving and receiving care has on the health and quality of life of care receivers and caregivers. While there have been many studies that associate caregivers' care work, financial difficulty, navigation, and other caregiving factors with family caregivers' psychological distress, we were interested not only in the factors related to family caregiver anxiety but also in hypothesizing how those effects occur. In this study, we used Andrew Hayes' PROCESS moderation analysis to explore the link between caregiver frailty, weekly care hours, and perceptions of financial difficulty, social support, and anxiety. In this analysis, we included 474 caregivers with relatively complete data on all of the variables. In regression analysis after controlling for gender and age, social loneliness (β = 0.245), frailty (β = 0.199), financial difficulty (β = 0.196), care time (β = 0.143), and navigation confidence (β = 0.131) were all significant. We then used PROCESS Model 6 to determine the significance of the direct, indirect, and total effects through the serial mediation model. The model pathway from frailty to care time to financial difficulty to social loneliness to anxiety was significant. The proportions of family caregivers who were moderately frail, anxious, and experiencing social loneliness after eighteen months of the COVID-19 pandemic found in this survey should be of concern to policymakers and healthcare providers.