Lighting virtual objects in a single image via coarse scene understanding

 CHEN XiaoWu, Jin Xin1 & WANG Ke

1Beijing Electronic Science and Technology Institute, Beijing 100070, China

Abstract

   Achieving convincing visual consistency between virtual objects and a real scene mainly relies on the lighting effects of virtual-real composition scenes. The problem becomes more challenging in lighting virtual objects in a single real image. Recently, scene understanding from a single image has made great progress. The estimated geometry, semantic labels and intrinsic components provide mostly coarse information, and are not accurate enough to re-render the whole scene. However, carefully integrating the estimated coarse information can lead to an estimate of the illumination parameters of the real scene. We present a novel method that uses the coarse information estimated by current scene understanding technology to estimate the parameters of a ray-based illumination model to light virtual objects in a real scene. Our key idea is to estimate the illumination via a sparse set of small 3D surfaces using normal and semantic constraints. The coarse shading image obtained by intrinsic image decomposition is considered as the irradiance of the selected small surfaces. The virtual objects are illuminated by the estimated illumination parameters. Experimental results show that our method can convincingly light virtual objects in a single real image, without any pre-recorded 3D geometry, reflectance, illumination acquisition equipment or imaging information of the image.

Gallery

lighting-1

Artistic illumination Transfer. (a) is the reference artistic painting. (b) is the input portrait. (c) is our transferred result. The illumination in (c) is similar to that of (a).

lighting-2

Overview of our system, which offers 2 main ways of transferring artistic illumination for portraits.

lighting-3

10 representative artistic portraits and artistic illumination templates. The 10 types are classified by artists from 500 artistic photographs and masterpiece paintings. They are featured mainly by the shadow and the light effects. The four basic shadow effects are: Triangle, Butterfly Loop and Split as described in [JZC*10]. With the addition of the side light, four other illumination types are: Butterfly plus Side, Loop plus side, Split plus Side, and Triangle plus Side. The other two types are: Bottom (with the key light from Bottom) and Three-quarters Side (with the key light generating three-quarters of the face in the light side). We collect 276 illumination templates covering these 10 artistic illumination types.

lighting-4

Artistic Illumination Template and the basic transfer workflow (choosing reference from our database). (a) is a reference artistic portrait:Mme Moitessier of the great French artist Jean Auguste Dominique Ingres; (b) is the average female face of 64 female face photos generated by the Beauty Check project; (c) is the shadow effects drawn by one painter; (d): light effects are added to (c). According to Eq.1, (e) and (f) are considered as the shadow template and the light template, respectively.The light template is linearly transformed for visualization. In our basic transfer workflow, the input face is aligned firstly. (g)and (h): 160 feature points in the average face and input portrait; (i) and (j) are the warped results of the shadow template (e)and light template (f), respectively. (k) is the transferred result with only shadow template applied. (l) is the final transferred result with both shadow and light effects.

lighting-5

The extended transfer workflow (User-provided Reference). (a) is the input portrait; (b) is the user-provided reference outside our database; (c) is the matched result obtained by our illumination matching method; (d) is the transferred result by referencing (c); (e) and (f) is the shadow template and the light template of (c); (g) and (h) are the warped templates from (e) and (f); (i) and (j) are the automatic adjusted templates from (g) and (h); (k) are the adjusted result of (d). Illumination in (k) is more similar to that of (b) than that of (c).

lighting-6

One Template, Multiple Inputs. With one of the reference portrait in our database, artistic illumination effects can be rendered in multiple styles of paintings, such as abstraction, cartoon, watercolor etc. Two examples are shown here. The input portraits in the first line are a portrait photo and its 2 abstraction styles generated by [ZZ10] (Abstract1) and [KLC09].

lighting-7

One Input, Multiple Templates. Multiple illumination effects can be transferred to one input portrait. Two categories of input portraits are show here, i.e. a painting and a computer generated painting(CGP) [KLC09]. The transferred results of fine art photos are shown in the multimedia attachments.

Paper and Slides

pdf

, , :Lighting virtual objects in a single image via coarse scene understanding. SCIENCE CHINA Information Sciences 57(9): 1-14 ().

pdf
, , :Single Image Based Illumination Estimation for Lighting Virtual Object in Real Scene. CAD/Graphics : 450-455
winrar    

BibTeX

@article{DBLP:journals/chinaf/ChenJW14,
  author    = {Xiaowu Chen and
               Xin Jin and
               Ke Wang},
  title     = {Lighting virtual objects in a single image via coarse scene understanding},
  journal   = {{SCIENCE} {CHINA} Information Sciences},
  volume    = {57},
  number    = {9},
  pages     = {1--14},
  year      = {2014},
  url       = {https://doi.org/10.1007/s11432-013-4936-0},
  doi       = {10.1007/s11432-013-4936-0},
  timestamp = {Wed, 17 May 2017 14:25:34 +0200},
  biburl    = {http://dblp.uni-trier.de/rec/bib/journals/chinaf/ChenJW14},
  bibsource = {dblp computer science bibliography, http://dblp.org}
}
@inproceedings{DBLP:conf/cadgraphics/ChenWJ11,
  author    = {Xiaowu Chen and
               Ke Wang and
               Xin Jin},
  title     = {Single Image Based Illumination Estimation for Lighting Virtual Object
               in Real Scene},
  booktitle = {12th International Conference on Computer-Aided Design and Computer
               Graphics, CAD/Graphics 2011, Jinan, China, September 15-17, 2011},
  pages     = {450--455},
  year      = {2011},
  crossref  = {DBLP:conf/cadgraphics/2011},
  url       = {https://doi.org/10.1109/CAD/Graphics.2011.19},
  doi       = {10.1109/CAD/Graphics.2011.19},
  timestamp = {Wed, 17 May 2017 10:54:59 +0200},
  biburl    = {http://dblp.uni-trier.de/rec/bib/conf/cadgraphics/ChenWJ11},
  bibsource = {dblp computer science bibliography, http://dblp.org}
}

发表评论

电子邮件地址不会被公开。 必填项已用*标注