ApoE4, a strong genetic risk factor for Alzheimer disease, has been associated with increased risk for severe COVID-19. However, it is unclear whether ApoE4 alters COVID-19 susceptibility or severity, and the role of direct viral infection in brain cells remains obscure. We tested the neurotropism of SARS-CoV2 in human-induced pluripotent stem cell (hiPSC) models and observed low-grade infection of neurons and astrocytes that is boosted in neuron-astrocyte co-cultures and organoids. We then generated isogenic ApoE3/3 and ApoE4/4 hiPSCs and found an increased rate of SARS-CoV-2 infection in ApoE4/4 neurons and astrocytes. ApoE4 astrocytes exhibited enlarged size and elevated nuclear fragmentation upon SARS-CoV-2 infection. Finally, we show that remdesivir treatment inhibits SARS-CoV2 infection of hiPSC neurons and astrocytes. These findings suggest that ApoE4 may play a causal role in COVID-19 severity. Understanding how risk factors impact COVID-19 susceptibility and severity will help us understand the potential long-term effects in different patient populations.