When diagnostic agreement is high, but reliability is low: some paradoxes occurring in joint independent neuropsychology assessments.

Two paradoxes can occur when neuropsychologists attempt to assess the reliability of a dichotomous diagnostic instrument (e.g., one measuring the presence or absence of Dyslexia or Autism). The first paradox occurs when two pairs of examiners both produce the same high level of agreement (e.g., 85%). Nonetheless, the level of chance-corrected agreement is relatively high for one pair (e.g., 70) and quite low for the other (e.g., .32). To illustrate the second paradox, consider two examiners who are in 80% agreement in their overall diagnosis of Dyslexia. Assume, further, that they are in 100% agreement in the proportion of cases they both diagnose as Dyslexic (20%) and as Non-Dyslexic (80%). Somewhat paradoxically, the level of chance-corrected interexaminer agreement for this pair of examiners calculates to only .37. In distinct contrast, a second set of examiners also in 80% overall agreement, is in appreciable disagreement with respect to diagnostic assignments. Thus, the first neuropsychologist: (a) classifies 65% of the cases as Non-Dyslexic, as opposed to 45% so diagnosed by the second neuropsychologist; and (b) classifies the remaining 35% as Dyslexic, as compared to the 55% so classified by the second examiner. Despite these phenomena, this second pair of examiners produces a much higher level of chance-corrected agreement than did the first pair, that is, a value of .61. The underlying reasons for both of these paradoxes, as well as their resolution, are presented.