The misuse of antibiotics as well as incorrect dosage or insufficient time for detoxification can result in the presence of pharmacologically active molecules in fresh milk. Hence, in many countries, commercially available milk has to be tested with immunological, chromatographic or microbiological analytical methods to avoid consumption of antibiotic residues. Here a novel, sensitive and portable assay setup for the detection and quantification of penicillin and kanamycin in whole fat milk (WFM) based on competitive magnetic immunodetection (cMID) is described and assay accuracy determined. For this, penicillin G and kanamycin-conjugates were generated and coated onto a matrix of immunofiltration columns (IFC). Biotinylated penicillin G or kanamycin-specific antibodies were pre-incubated with antibiotics-containing samples and subsequently applied onto IFC to determine the concentration of antibiotics through the competition of antibody-binding to the antibiotic-conjugate molecules. Bound antibodies were labeled with streptavidin-coated magnetic particles and quantified using frequency magnetic mixing technology. Based on calibration measurements in WFM with detection limits of 1.33 ng·mL for penicillin G and 1.0 ng·mL for kanamycin, spiked WFM samples were analyzed, revealing highly accurate recovery rates and assay precision. Our results demonstrate the suitability of cMID-based competition assay for reliable and easy on-site testing of milk.