Drug delivery is a key component of nanomedicine, and conventional delivery relies on the adsorption or encapsulation of drug molecules to a nanomaterial. Many delivery vehicles contain metal ions, such as metal-organic frameworks, metal oxides, transition metal dichalcogenides, MXene, and noble metal nanoparticles. These materials have a high metal content and pose potential long-term toxicity concerns leading to difficulties for clinical approval. In this review, recent developments are summarized in the use of drug molecules as ligands for metal coordination forming various nanomaterials and soft materials. In these cases, the drug-to-metal ratio is much higher than conventional adsorption-based strategies. The drug molecules are divided into small-molecule drugs, nucleic acids, and proteins. The formed hybrid materials mainly include nanoparticles and hydrogels, upon which targeting ligands can be grafted to improve efficacy and further decrease toxicity. The application of these materials for addressing cancer, viral infection, bacterial infection inflammatory bowel disease, and bone diseases is reviewed. In the end, some future directions are discussed from fundamental research, materials science, and medicine.