Monitor

Consumer electronics: New approaches to photography treat it as a branch of computing as well as optics, making possible a range of new tricks

The Economist, Sep 3rd 2011 | from the print edition

http://www.economist.com/node/21527019PHOTOGRAPHY can trace its roots to the camera obscura, the optical principles of which were understood as early as the 5th century BC. Latin for a darkened chamber, it was just that: a shrouded box or room with a pinhole at one end through which light from the outside was projected onto a screen inside, displaying an inverted image. This, you might think, is a world away from modern digital cameras, brimming with fancy electronics which capture the wavelengths and intensity of light to produce high-resolution digital files. But the basic idea of focusing rays through an aperture onto a two-dimensional surface remains the same.

Now a novel approach to photographic imaging is making its way into cameras and smartphones. Computational photography, a subdiscipline of computer graphics, does not simply capture single images. The basic premise is to use multiple exposures, or multiple lenses, to capture information from which photographs may be derived. These data contain myriad potential pictures which software then converts into what looks like a conventional photo. More computer animation than pinhole camera, in other words, though using real light refracted through a lens rather than the virtual sort.

In this Technology Quarterly

Reprints

Related topics

The best known example of computational photography is high-dynamic-range (HDR) imaging, which combines multiple photos shot in rapid succession, and at different exposures, into one picture of superior quality. Where a single snap may miss out on detail in the lightest and darkest areas, an HDR image of the same scene looks preternaturally well lit (see above). HDR used to be a specialised technique employed mostly by professionals. That changed when Apple added it as an option in the iPhone 4. (Previous iPhones lacked the oomph to crunch relevant data quickly enough to be practical.) Other examples include cameras and apps that stitch together panoramas from overlapping images shot in an arc around a single point or as a moving sequence.

But HDR and panoramas are just two ways to splice together images of the same subject, says Marc Levoy of Stanford University, who kickstarted computational photography in a paper he wrote in 1996 with his colleague Pat Hanrahan. Since then aspects of the field have moved from academia into commercial products. This, Dr Levoy explains, is mainly down to the processing power of devices, such as camera-equipped smartphones, growing faster than the capacity of sensors which record light data. “You are getting more computing power per pixel,” he says.

To show off the potential of some new techniques, Dr Levoy created the SynthCam app for the iPhone and other Apple devices. The app takes a number of successive video frames and processes them into a single, static image that improves on the original in a variety of ways. He and his colleagues have also built several models of Frankencamera, using bits of kit found in commercially available devices—both low-end, like inexpensive cell phones, and high-end, such as pricey single-lens reflex cameras. The Frankencameras use a host of clever algorithms to capture sequences of images and turn them into better photos. SynthCam and Frankencameras can improve the quality of pictures taken in low-light conditions, which are usually quite grainy. They can also create an artificial focus that is absent from the original images, or render a foreground figure in crisp focus against a blurred backdrop.

Point and shoot

Still, for all the superior results that computational photography offers, Dr Levoy laments, camera-makers have been loth to embrace its potential. This is about to change. In June this year Ren Ng, a former student of Dr Levoy’s at Stanford, launched a new company called Lytro, promising to start selling an affordable snapshot camera later this year.

Rather than use conventional technology, as the Frankencamera does, to meld successive exposures, Dr Ng has figured out a way to capture lots of images simultaneously. This approach is known as light-field photography, and Lytro’s camera will be its first commercial incarnation. In physics, a light field describes the direction of all the idealised light rays passing through an area. Dr Levoy’s and Dr Hanrahan’s seminal paper described a new way to model this field mathematically. Now, 15 years later, Dr Ng has worked out how to implement the technique using off-the-shelf chips.

Dr Ng’s camera uses an array of several hundred thousand microlenses inserted between an ordinary camera lens and digital image sensor. Each microlens functions as a kind of superpixel. A typical camera works by recording where light strikes the focal plane—the area onto which rays passing through a lens are captured. In traditional cameras the focal plane was a piece of film; modern ones use arrays of digital sensors. In Lytro’s case, however, a light ray passes through the main lens—which uses a wide aperture—and then through one of the microlenses. Only then does it hit a sensor. By calculating the path between the microlens and the sensor, the precise direction of a light ray can be reconstructed. That in turn means it is possible to determine where the ray would have struck if the focal plane had been moved. Moving the focal plane is equivalent to refocusing the lens, so any point in the light field can then, in effect, be brought into sharp focus. In other words, images can be refocused after they have been taken (see below).

Shoot first, refocus later

It is also possible to fiddle with an image’s depth of field—photographic jargon for the space between the closest and most distant points in an image that are in focus—or even create an image in which every point is in focus. And the light-field approach can produce a compelling simulation of a stereoscopic image. Lytro’s website lets visitors fiddle with existing images to see how some of these features will work.

Unlike the information it records, the camera itself is simple. The main lens is fixed in place; there is no auto-focus, auto-aperture, or other machinery which needs to be activated every time a photo is taken. Such adjustments take time, causing a lag between pressing the shutter-release button and actually capturing the image. Lytro’s snaps, by contrast, will be truly instantaneous, just as they were in old snapshot cameras. And, since the lens is preset always to capture the greatest amount of light possible, exposure time can be short, even in poorly lit conditions.

One potential downside is the camera’s low resolution, which is defined by the number of microlenses, because the processing software treats each microlens as a single pixel. The sample images on Lytro’s website measure 525 by 525 pixels, which works perfectly well online, but will not pass muster in print. This might not matter, though. Nowadays people make fewer photographic prints, especially large-format ones where resolution counts. By contrast, billions of photographs are shared online each year. Professional photographers may still seek higher pixel counts, but there is no reason why future versions of the device could not offer more microlenses.

For now, though, Lytro is targeting internet photo-sharers. It will let owners of its camera upload the image data and the processing tools to Facebook and other social networks. The firm has reportedly already raised $50m. Investors must be hoping that consumers find all the irritants that Lytro’s camera removes, like blurred or dim pictures, niggling enough to want them eliminated once and for all from their holiday snaps.