The study of observer variation in the radiological classification of pneumoconiosis.

In a long-term investigation such as the National Coal Board's Pneumoconiosis Field Research (P.F.R.), it is essential to establish satisfactory and stable procedures for making the necessary observations and measurements. It is equally important regularly to apply suitable methods of checking the accuracy and consistency of the various observations and measurements. One aspect of vital importance in the P.F.R. is the classification of the series of chest radiographs taken, at intervals, of all the men under observation. This is inevitably a subjective process, and (as with other similar fields of work) it is desirable to obtain some understanding of the basic process behind the operation. This can usefully be done by the help of “models” designed to describe the process, if necessary in simplified terms. The problem of the radiological classification of pneumoconiosis has been studied hitherto in terms of coefficients of disagreement (inter-observer variation) and inconsistency (intra-observer variation), but for various reasons the method was not considered entirely satisfactory. New methods of approach were therefore developed for studying the performance of the two doctors responsible for the film reading in the Research, and two distinct “models” were derived. The advantages and disadvantages of each are described in the paper, together with the applications of the two models to the study of some of the problems arising in the course of the investigation. The first model is based on the assumption that if a film is selected at random from a batch representing a whole colliery population, and that if the film is of “true” category i, the chance of its being read as another category (j) is a constant, P, which depends upon the observer concerned, the particular batch of films being read, and the values of i and j. This model enables the performance of the readers to be monitored satisfactorily, and it has also been used to investigate different methods for arriving at an agreed, or “definitive”, assessment of radiological abnormality. The P model suffers from the disadvantage of applying only to “average” films, and the assumptions made are such that it manifestly does not provide an entirely realistic representation of the reading process on any particular film. The second “improved” model was therefore developed to overcome this criticism. Briefly, it is considered that each film is representative of a unique degree of abnormality, located on a continuum, or abnormality scale, which covers the whole range of simple pneumoconiosis. The scale of abnormality is then chosen in such a way that, whatever the true degree of abnormality of the film, the observer's readings will be normally distributed about the true value with constant bias and variability at all points along the scale. The very large number of readings available has been analysed to determine the optimum positions of the category boundaries on the abnormality scale and in this way the scale has been unambiguously defined. The model enables the routine reading standards to be monitored, and it has also been used to investigate the underlying distribution of abnormality at individual collieries. Its chief disadvantage is the extensive computational work required. The “fit” of both models to the data collected in the Research is shown to be satisfactory and on balance it appears that both have applications in this field of study. The method chosen in any given circumstance will depend upon the particular requirement and the facilities available for computational work.