Multimedia Systems Computer Networks and Communications
Evaluation of image person removal tasks: when data synthesis encounters 3D reconstruction
Abstract
Data synthesis has become an important means of addressing the shortage of data for training deep networks, especially when image–label pairs are hard to collect. This problem is particularly evident in the image person removal task, where images with and without a specific person cannot be captured at the same time. This paper therefore proposes a method for generating synthetic datasets for this task. Building on our previous work, we take a step further by combining 3D reconstruction with data synthesis to improve the diversity and realism of the data. Specifically, we design a pipeline that builds a 3D library of real people from multi-view images of real scenes using multi-view stereo (MVS). The library contains diverse appearances and body motions and can be extended. Lighting intensity and orientation are discretized and selected by a brute-force search that minimizes the discrepancy in the background region, and image harmonization is then applied to improve the synthesized images. In the experiments, rather than evaluating only the proposed method, we conduct a cross-domain evaluation of image person removal together with our previous work. The conclusions are informative for both this task and data synthesis in general. As far as we know, this is the first study to fully evaluate data synthesis on such a task.