Computational Imaging

Handheld Multi-Frame Super-Resolution TOG.

  Compared to DSLR cameras, smartphone cameras have smaller sensors, which limits their spatial resolution; smaller apertures, which limits their light gathering ability; and smaller pixels, which reduces their signal-tonoise ratio. The use of color filter arrays (CFAs) requires demosaicing, which further degrades resolution. In this paper, we supplant the use of traditional demosaicing in single-frame and burst photography pipelines with a multiframe super-resolution algorithm that creates a complete RGB image directly from a burst of CFA raw images. We harness natural hand tremor, typical in handheld photography, to acquire a burst of raw frames with small offsets. These frames are then aligned and merged to form a single image with red, green, and blue values at every pixel site. This approach, which includes no explicit demosaicing step, serves to both increase image resolution and boost signal to noise ratio. Our algorithm is robust to challenging scene conditions: local motion, occlusion, or scene changes. It runs at 100 milliseconds per 12-megapixel RAW input burst frame on mass-produced mobile phones. Specifically, the algorithm is the basis of the Super-Res Zoom feature, as well as the default merge method in Night Sight mode (whether zooming or not) on Google’s flagship phone.

Image Stylization: From Predefined to Personalized IET Computer VIsion

  We present a framework for interactive design of new image stylizations using a wide range of predefined filter blocks. Both novel and off-the-shelf image filtering and rendering techniques are extended and combined to allow the user to unleash their creativity to intuitively invent, modify, and tune new styles from a given set of filters. In parallel to this manual design, we propose a novel procedural approach that automatically assembles sequences of filters, leading to unique and novel styles. An important aim of our framework is to allow for interactive exploration and design, as well as to enable videos and camera streams to be stylized on the fly. In order to achieve this real-time performance, we use the "Best Linear Adaptive Enhancement" (BLADE) framework -- an interpretable shallow machine learning method that simulates complex filter blocks in real time. Our representative results include over a dozen styles designed using our interactive tool, a set of styles created procedurally, and new filters trained with our BLADE approach.

Urban Procedural Models: Inverse Design and Traffic

Designing Large-Scale Interactive Traffic Animations for Urban Modeling EG.

  We presented an approach to interactively "paint" a desired vehicular traffic behavior and animation and then the system automatically computes a realistic 3D urban model yielding the specified behavior. We used our system to control traffic behaviors such as road occupancy, travel time, and CO emission. Our framework includes a novel traffic microsim- ulation approach which yields the high performance needed for our interactive design tool. Our traffic manipulation strategy adapts a MCMC method to explore the solution space by performing a set of road network changes.

Inverse Design of Urban Procedural Models TOG

  We have coupled an automatic inverse design approach for urban procedural modeling with forward procedural modeling. Urban indicators are intuitive metrics for measuring the desirability of urban areas. The relationship of indicators to the procedural model is in general unknown and complex which has until now hindered their direct specification. We tackle the well-known open problem of controlling procedural modeling by providing a generalized mechanism that allows users to specify arbitrary target indicators and automatically compute the optimal parameters to obtain the desired output.