A major challenge in effective cancer treatment is the variability of drug responses among patients. Patient-derived organoids greatly preserve the genetic and histological characteristics even the drug sensitivities of primary tumor tissues, therefore provide a compelling approach to predict clinical outcome. However, the individual organoid culture and following drug response test are time and cost-consuming, which hinders the potential clinical application. Here, we developed PharmaFormer, a clinical drug response prediction model based on custom Transformer architecture and transfer learning. PharmaFormer was initially pre-trained with the abundant gene expression and drug sensitivity data of 2D cell lines, and was then finalized through a model further fine-tuned with the limited organoid pharmacogenomic data accumulated at the present stage. Our results demonstrate that PharmaFormer, integrating both pan-cancer cell lines and organoids of a specific type of tumor, provides a dramatically improved accurate prediction of clinical drug response. This study highlights that advanced AI models combined with biomimetic organoid models will accelerate precision medicine and future drug development.