BACKGROUND: Shotgun metagenomics has been used clinically for diagnosing infectious diseases. However, most technical assessments have been limited to individual sets of reference standards, experimental workflows, and laboratories. METHODS: A reference panel and performance metrics were designed and used to examine the performance of shotgun metagenomics at 17 laboratories in a coordinated collaborative study. We comprehensively assessed the reliability, key performance determinants, reproducibility, and quantitative potential. FINDINGS: Assay performance varied significantly across sites and microbial classes, with a read depth of 20 millions as a generally cost-efficient assay setting. Results of mapped reads by shotgun metagenomics could indicate relative and intra-site (but not absolute or inter-site) microbial abundance. INTERPRETATION: Assay performance was significantly impacted by the microbial type, the host context, and read depth, which emphasizes the importance of these factors when designing reference reagents and benchmarking studies. Across sites, workflows and platforms, false positive reporting and considerable site/library effects were common challenges to the assay's accuracy and quantifiability. Our study also suggested that laboratory-developed shotgun metagenomics tests for pathogen detection should aim to detect microbes at 500 CFU/mL (or copies/mL) in a clinically relevant host context (10^5 human cells/mL) within a 24h turn-around time, and with an efficient read depth of 20M. FUNDING: This work was supported by National Science and Technology Major Project of China (2018ZX10102001).