Daily standardization of routinely collected milk mid-infrared spectra from dairy herd improvement testing in a statistical framework.

Supported by analyses of standard milk samples with known reference values, mid-infrared (MIR) spectroscopic analysis of milk is characterized by high accuracy and repeatability, particularly for the main milk components. In laboratory routines, this is assisted by slope-intercept correction procedures, where post-measurement corrections of the predictions of regular samples are performed. Independently of this, deviations and drifts can be observed in MIR spectra, both across instruments and over time. The aim of this study was to demonstrate an innovative approach for the standardization of MIR spectra on a daily level using only statistical tools with an existing complex historical dataset. In the underlying procedure, a framework of regression models considering results of laboratory analyses of milk and information on the animal, such as DIM and parity, are used to estimate daily-, instrument- and wavenumber-wise standardization coefficients based on routine DHI data. Data from the first half of 2022 were provided by the Landeskontrollverband Niedersachsen e.V. (Leer, Germany) and comprised 2.3 million spectra from 5 FOSS (Hillerød, Denmark) instruments as well as the corresponding DHI data (dataset I). Additionally, multiple analyses of 5,335 DHI samples on 3 instruments were carried out (dataset II). Furthermore, 60,961 analyses of standard milks were available (dataset III). Dataset I was used to estimate the standardization coefficients. To investigate the effects of standardization on model calibration, dataset II was used to build fat prediction models using both raw and standardized spectra. In the statistical analysis, dataset II was used for principal component analyses and an inter-instrument comparison of the spectra as well as for comparison of fat predictability with and without standardization. Dataset III was used to compare fat reference values of standard milks with MIR-based laboratory results and own predictions using both raw and standardized spectra. In dataset II, the developed standardization led to a harmonization of spectra and predictions and corrected both general and temporary instrument effects. During fat model calibration, the predictability and model transferability across instruments were improved. The root mean squared error (RMSE) of a forward-in-time validation decreased from 0.0398 with raw to 0.0246% fat with standardized spectra. When comparing the reference values of standard milks with predictions based on raw and standardized spectra of dataset III, the RMSE was reduced from 0.0406% to 0.0195% fat due to standardization, even with transferring the models to external data of other instruments. This study highlights the need for frequent standardization both across and within instruments over time. To handle this, the proposed procedure featuring a daily standardization seems to be a promising approach. Under the given laboratory conditions and farm structures (i.e., DHI data from Holstein cows and spectra from FOSS Instruments), this work can be regarded as a proof of concept. The influence of standardization was demonstrated both at the level of the spectra and at the level of the predictions using the example of milk fat. A big advantage is that it is a software-based solution that can be easily modified and scaled in terms of its application.