V. Sitzmann, S. Diamond, Y. Peng, X. Dun, S. Boyd, W. Heidrich, F. Heide, G. Wetzstein
ACM Transactions on Graphics (Proc. SIGGRAPH), (2018)
In typical cameras the optical system is designed first; once it is
fixed, the parameters in the image processing algorithm are tuned to get
good image reproduction. In contrast to this sequential design
approach, we consider joint optimization of an optical system (for
example, the physical shape of the lens) together with the parameters of
the reconstruction algorithm. We build a fully-differentiable
simulation model that maps the true source image to the reconstructed
one. The model includes diffractive light propagation, depth and
wavelength-dependent effects, noise and nonlinearities, and the image
post-processing. We jointly optimize the optical parameters and the
image processing algorithm parameters so as to minimize the deviation
between the true and reconstructed image, over a large set of images. We
implement our joint optimization method using autodifferentiation to
efficiently compute parameter gradients in a stochastic optimization
algorithm. We demonstrate the efficacy of this approach by applying it
to achromatic extended depth of field and snapshot super-resolution
imaging.