MaterialGAN: Reflectance Capture using a Generative SVBRDF Model
Yu Guo1, Cameron Smith2, Miloš Hašan2, Kalyan Sunkavalli2, and Shuang Zhao1
1University of California, Irvine     2Adobe Research
ACM Transactions on Graphics (SIGGRAPH Asia 2020), 39(6), 2020
teaser
Abstract

We address the problem of reconstructing spatially-varying BRDFs from a small set of image measurements. This is a fundamentally under-constrained problem, and previous work has relied on using various regularization priors or on capturing many images to produce plausible results. In this work, we present MaterialGAN, a deep generative convolutional network based on StyleGAN2, trained to synthesize realistic SVBRDF parameter maps. We show that MaterialGAN can be used as a powerful material prior in an inverse rendering framework: we optimize in its latent representation to generate material maps that match the appearance of the captured images when rendered. We demonstrate this framework on the task of reconstructing SVBRDFs from images captured under flash illumination using a hand-held mobile phone. Our method succeeds in producing plausible material maps that accurately reproduce the target images, and outperforms previous state-of-the-art material capture methods in evaluations on both synthetic and real data. Furthermore, our GAN-based latent space allows for high-level semantic material editing operations such as generating material variations and material morphing.

Paper Video
Downloads
  • Paper: pdf (56 MB)
  • Supplemental material: html, zip (264 MB)
  • Code & data: coming soon
Bibtex Citation
@article{Guo:2020:MaterialGAN,
  title={MaterialGAN: Reflectance Capture using a Generative SVBRDF Model},
  author={Guo, Yu and Smith, Cameron and Ha\v{s}an, Milo\v{s} and Sunkavalli, Kalyan and Zhao, Shuang},
  journal={ACM Trans. Graph.},
  volume={39},
  number={6},
  year={2020},
  pages={254:1--254:13}
}
Acknowledgments

We are grateful to the anonymous reviewers for their help in improving this work. This research was started during Yu Guo's internship at Adobe Research. We thank TJ Rhodes for help with material capture hardware setup. This work was supported in part by NSF IIS-1813553.