Accelerated use of telemedicine during the COVID-19 pandemic enabled uninterrupted healthcare delivery while unmasking care disparities for several vulnerable communities. The social determinants of health (SDOH) serve as a critical model for understanding how the circumstances in which people are born, work, and live impact health outcomes. We performed semi-structured interviews to understand patients and providers' experiences with telemedicine encounters during the COVID-19 pandemic. Through a deductive approach, we applied the SDOH to determine telemedicine's role and impact within this framework. Overall, patient and provider interviews supported the use of existing SDOH domains to describe disparities in Internet access and telemedicine use, rather than reframing technology as a sixth SDOH. In order to mitigate the digital divide, we identify and propose solutions that address SDOH-related barriers that shape the use of health information technologies.