A Comprehensive Introduction
Physics-based rendering algorithms generate photorealistic images by simulating the flow of light through a detailed mathematical representation of a virtual scene. In contrast, physics-based differentiable rendering algorithms focus on computing derivative of images exhibiting complex light transport effects (e.g., soft shadows, interreflection, and caustics) with respect to arbitrary scene parameters such as camera pose, object geometry (e.g., vertex positions) as well as spatially varying material properties expressed as 2D textures and 3D volumes. This new level of generality has made physics-based differentiable rendering a key ingredient for solving many challenging inverse-rendering problems, that is, the search of scene configurations optimizing user-specified objective functions, using gradient-based methods
Further, these techniques can be incorporated into probabilistic inference and machine learning pipelines. For instance, differentiable renderers allow "rendering losses" to be computed with complex light transport effects captured. Additionally, they can be used as generative models that synthesize photorealistic images.
Challenges. Compared to its "ordinary" counterpart, physics-based differentiable rendering introduces unique theoretical and practical challenges. For instance, practical problems can involve many (e.g., 106--1010) parameters, making simple techniques for differentiation such as finite differences impractical. More advanced tools for automatic differentiation (e.g., PyTorch or Tensorflow) record a graph of intermediate computation steps that tends to become prohibitively large. Lastly, geometric derivatives involve a unique challenge: boundaries of objects introduce troublesome discontinuities during the computation of shadows and interreflections that lead to incorrect gradients if precautions are not taken. Thankfully, recent advances in physics-based differentiable rendering theory have enabled the differentiation of radiometric measurements with respect to arbitrary scene parameters as well as unbiased Monte Carlo estimators. In this course, we provide an in-depth introduction to general-purpose physics-based differentiable rendering.