This study examines the effects of noise and the use of an Intelligent Virtual Assistant (IVA) on the task performance and workload of office workers. Data were collected from forty-eight adults across varied office task scenarios (i.e., sending an email, setting up a timer/reminder, and searching for a phone number/address) and noise types (i.e., silence, non-verbal noise, and verbal noise). The baseline for this study is measured without the use of an IVA. Significant differences in performance and workload were found on both objective and subjective measures. In particular, verbal noise emerged as the primary factor affecting performance using an IVA. Task performance was dependent on the task scenario and noise type. Subjective ratings found that participants preferred to use IVA for less complex tasks. Future work can focus more on the effects of tasks, demographics, and learning curves. Furthermore, this work can help guide IVA system designers by highlighting factors affecting performance.