HL235 is a new cathepsin K inhibitor designed and synthesized to treat osteoporosis. Since HL235 has poor aqueous solubility, a self-microemulsifying drug delivery system (SMEDDS) was formulated to enhance its oral bioavailability. A solubility study of HL235 was performed to select a suitable oil, surfactant and cosurfactant. Pseudoternary phase diagrams were plotted to identify the microemulsion region and to determine the range of components in the isotropic mixture. D-optimal mixture design and a desirability function were introduced to optimize the SMEDDS formulation for the desired physicochemical characteristics, i.e., high drug concentration at 15 min after dilution with simulated gastric fluid (SGF) and high solubilization capacity. The optimized HL235-loaded SMEDDS formulation consisted of 5.0% Capmul MCM EP (oil), 75.0% Tween 20 (surfactant) and 20.0% Carbitol (cosurfactant). The droplet size of the microemulsion formed by the optimized formulation was 10.7 ± 1.6 nm, and the droplets were spherical in shape. Pharmacokinetic studies in rats showed that the relative oral bioavailability of the SMEDDS formulation increased up to 3.22-fold compared to its solution in DMSO:PEG400 (8:92, v/v). Thus, the formulation of SMEDDS optimized by D-optimal mixture design could be a promising approach to improve the oral bioavailability of HL235.