BACKGROUND: The penicillin adverse drug reaction (ADR) label is common in electronic health records (EHRs). However, there is significant misclassification between allergy and intolerance within the EHR and most patients can be delabelled after an immunologic assessment. Machine learning natural language processing may be able to assist with the categorisation and risk stratification of penicillin ADRs. OBJECTIVE: The aim of this study was to use text entered into an EHR to derive and evaluate machine learning models to classify penicillin ADRs and assess the risk of true allergy. METHODS: Machine learning natural language processing was applied to free-text penicillin ADR data extracted from a public health system EHR. The model was developed by training on labelled dataset. ADR entries were split into training and testing datasets and used to develop and test a variety of machine learning models. These were compared to categorisation with a simple algorithm using keyword search. RESULTS: The best performing model for the classification of penicillin ADRs as being consistent with allergy or intolerance was the artificial neural network (AUC 0.994, sensitivity 0.99, specificity 0.96). The artificial neural network also achieved the highest AUC in the classification of high- or low-risk of true allergy (AUC 0.988, sensitivity 0.99, specificity 0.99). All ADR labels were able to be classified using these machine learning models, whereas a small proportion were unclassifiable using the simple algorithm as they contained no keywords. CONCLUSION: Machine learning natural language processing performed similarly to expert criteria in classifying and risk stratifying penicillin ADRs labels. These models outperformed simpler algorithms in their ability to interpret free-text data contained in the EHR. The automated evaluation of penicillin ADR labels may allow real-time risk stratification to facilitate delabelling and improve the specificity of prescribing alerts.