Differentiable Time-Gated Rendering

Supplemental Materials

Lifan Wu^1*, Guangyan Cai^2*, Ravi Ramamoorthi³, and Shuang Zhao^{2 1}NVIDIA        ²University of California, Irvine        ³University of California, San Diego        (* equal contribution)

In this material, we show several synthetic inverse-rendering results. For each example, we minimize the image loss (measured in RMSE) between input (steady-state or ToF) target images and renderings. Additionally, we use parameter RMSE to evaluate the quality of inverse-rendering results. This information is not used by the optimizations.

In what follows, we demonstrate the necessity of supporting global interreflection and the practical advantage of our method over finite differences by comparing inverse-rendering performances.

This Corridor scene includes a two-segment corridor with the camera and an area light located at the opening of the corridor. Taking as input 20 target ToF images, we search for global translations in x- and z-directions of the four vertices at the end of the second segment.

We use the Adam method to solve the inverse-rendering optimizations with identical initial configurations and learning rates for all methods. We also adjust the sample count so that each iteration takes approximately equal time for all methods.

3-bounce interreflection

When limiting the maximal number of reflections to three, the resulting time-gated renderings suffer high energy loss, causing the optimization to diverge.

Full interreflection (finite differences)

When using full interreflections with derivatives computed with finite differences, the forward renderings are accurate. Unfortunately, the optimization still fails to find the correct solution due to the bias and high variance in FD gradient estimates.

Full interreflection (ours)

Using full interreflections and derivatives estimated using our method, the inverse-rendering optimization successfully finds the correct result.

We now demonstrate the effectiveness of our ellipsoidal next-event estimation and antithetic sampling for handling near-delta path importance functions.

This Cube scene contains a glossy cube lit by an area light. We search for the vertical position of the cube using four target ToF images with near-delta path importance functions.

We use identical initial configurations, learning rates, and time per iteration for all methods.

Standard

Without ellipsoidal NEE or antithetic sampling, the standard estimator suffers from very high variance, causing the optimization to diverge.

Ellipsoidal NEE + antithetic sampling

With our ellipsoidal NEE and antithetic sampling, significant variance reduction can be achieved, allowing the optimization to converge easily.

We now compare inverse-rendering performance using steady-state and time-gated images. In this example, we have a diffuse surface lit and viewed from above. We search for the vertical positions of the 121 vertices of this surface. We use the Adam method with identical initial configurations to solve both optimizations.

Steady-state

When using one steady-state target image, the problem becomes highly under-constrained, causing the inverse-rendering optimization to get stuck at a local minimum.

Time-gated

When using 12 time-gated images (from the same camera location), the added information allows the optimization to converge at the correct solution.

Branches

• This example involves a tree-like object lit by an area light, casting shadows on the ground. We search for the rotation angle of a tree-like object, so that the difference (measured in image RMSE) between time-gated renderings of the shadow and the target time-of-flight (ToF) images is minimized.

Bunny

• This example consists of a diffuse box that contains a rough-glass bunny lit by a small area light. Both the bunny and the light are blocked from the camera’s point of view, mimicking a non-line-of-sight (NLOS) imaging configuration. We optimize the translation of the bunny (along the x- and y-axis).

Sofa

• This example contains a sofa-like diffuse object lit by a large area light. The object is outside the time-of-flight camera's field of view. We optimize the surface albedo and the shape of the object jointly.

Tree

• This example has a glossy tree lit by a small area light with a near-delta time gate. We search for the horizontal location of the light.