The auricular branch of the vagus nerve runs superficially, which makes it a favorable target for non-invasive stimulation techniques to modulate vagal activity. For this reason, there have been many early-stage clinical trials on a diverse range of conditions. These trials often report conflicting results for the same indication. Using the Cochrane Risk of Bias tool we conducted a systematic review of auricular vagus nerve stimulation (aVNS) randomized controlled trials (RCTs) to identify the factors that led to these conflicting results. The majority of aVNS studies were assessed as having "some" or "high" risk of bias, which makes it difficult to interpret their results in a broader context. There is evidence of a modest decrease in heart rate during higher stimulation dosages, sometimes at above the level of sensory discomfort. Findings on heart rate variability conflict between studies and are hindered by trial design, including inappropriate washout periods, and multiple methods used to quantify heart rate variability. There is early-stage evidence to suggest aVNS may reduce circulating levels and endotoxin-induced levels of inflammatory markers. Studies on epilepsy reached primary endpoints similar to previous RCTs testing implantable vagus nerve stimulation therapy. Preliminary evidence shows that aVNS ameliorated pathological pain but not evoked pain. Based on results of the Cochrane analysis we list common improvements for the reporting of results, which can be implemented immediately to improve the quality of evidence. In the long term, existing data from aVNS studies and salient lessons from drug development highlight the need for direct measures of local neural target engagement. Direct measures of neural activity around the electrode will provide data for the optimization of electrode design, placement, and stimulation waveform parameters to improve on-target engagement and minimize off-target activation. Furthermore, direct measures of target engagement, along with consistent evaluation of blinding success, must be used to improve the design of controls-a major source of concern identified in the Cochrane analysis. The need for direct measures of neural target engagement and consistent evaluation of blinding success is applicable to the development of other paresthesia-inducing neuromodulation therapies and their control designs.