IEEE Trans Vis Comput Graph. 2017 Nov;23(11):2447-2454. doi: 10.1109/TVCG.2017.2734425. Epub 2017 Aug 11.
We present a novel real-time approach for user-guided intrinsic decomposition of static scenes captured by an RGB-D sensor. In the first step, we acquire a three-dimensional representation of the scene using a dense volumetric reconstruction framework. The obtained reconstruction serves as a proxy to densely fuse reflectance estimates and to store user-provided constraints in three-dimensional space. User constraints, in the form of constant shading and reflectance strokes, can be placed directly on the real-world geometry using an intuitive touch-based interaction metaphor, or using interactive mouse strokes. Fusing the decomposition results and constraints in three-dimensional space allows for robust propagation of this information to novel views by re-projection. We leverage this information to improve on the decomposition quality of existing intrinsic video decomposition techniques by further constraining the ill-posed decomposition problem. In addition to improved decomposition quality, we show a variety of live augmented reality applications such as recoloring of objects, relighting of scenes and editing of material appearance.
我们提出了一种新的实时方法，用于引导用户对由 RGB-D 传感器捕获的静态场景进行内在分解。在第一步中，我们使用密集体积重建框架获取场景的三维表示。所获得的重建用作密集融合反射率估计的代理，并将用户提供的约束存储在三维空间中。用户约束（例如恒定阴影和反射笔划）可以使用直观的基于触摸的交互隐喻直接放置在真实世界的几何形状上，或者使用交互式鼠标笔划。在三维空间中融合分解结果和约束可以通过重新投影来稳健地将此信息传播到新视图。我们利用此信息通过进一步约束病态分解问题来提高现有内在视频分解技术的分解质量。除了提高分解质量外，我们还展示了各种实时增强现实应用，例如物体重新着色、场景重新照明和材料外观编辑。
