Mobile- and web-based psychological research are a valuable addition to the set of tools available for scientific study, reducing logistical barriers for research participation and allowing the recruitment of larger and more diverse participant groups. However, this comes at the cost of reduced control over the technology used by participants, which can introduce new sources of variability into study results. In this study, we examined differences in measured performance on timed and untimed cognitive tests between users of common digital devices in 59,587 (Study 1) and 3818 (Study 2) visitors to TestMyBrain.org , a web-based cognitive testing platform. Controlling for age, gender, educational background, and cognitive performance on an untimed vocabulary test, users of mobile devices, particularly Android smartphones, showed significantly slower performance on tests of reaction time than users of laptop and desktop computers, suggesting that differences in device latency affect measured reaction times. Users of devices that differ in user interface (e.g. screen size, mouse vs. touchscreen) also show significant differences (p < 0.001) in measured performance on tests requiring fast reactions or fine motor movements. By quantifying the contribution of device differences to measured cognitive performance in an online setting, we hope to improve the accuracy of mobile- and web-based cognitive assessments, allowing these methods to be used more effectively.