Single-cell RNA sequencing (scRNA-seq) massively profiles transcriptomes of individual cells encapsulated in barcoded droplets in parallel. However, in real-world scRNA-seq data, many barcoded droplets do not contain cells, but instead, they capture a fraction of ambient RNAs released from damaged or lysed cells. A typical first step to analyze scRNA-seq data is to filter out cell-free droplets and isolate cell-containing droplets, but distinguishing them is often challenging; incorrect filtering may mislead the downstream analysis substantially. We propose SiftCell, a suite of software tools to identify and visualize cell-containing and cell-free droplets in manifold space via randomization (SiftCell-Shuffle) to classify between the two types of droplets (SiftCell-Boost) and to quantify the contribution of ambient RNAs for each droplet (SiftCell-Mix). By applying our method to datasets obtained by various single-cell platforms, we show that SiftCell provides a streamlined way to perform upstream quality control of scRNA-seq, which is more comprehensive and accurate than existing methods.