An Eye-Opening Look at a New Approach to Image Processing
How biology-based advancements in image processing are poised to transform the video industry by significantly improving compression and quality.
In 1966, a little company called Dolby was about to revolutionize the audio industry. The company’s founder, Ray Dolby, had invented a new form of audio compression and expansion technology and was about to commercialize it. The technology dramatically reduced the background hissing noise heard in all professional tape recordings. And it did not cause any unwanted side effects in the material being recorded.
The company manufactured a noise reduction device and marketed it to record companies, and the rest is history. Without this technology, multitrack recordings would never have been possible, and neither would the surround sound you hear in thousands of cinemas and millions of home theaters today.
In 2009, some 43 years after Dolby’s noise reduction technology transformed the music industry, the same thing is about to happen again – this time, in video. And it all came about “almost by accident,” says Sean McCarthy, Ph.D. and fellow of the technical staff and network video solutions at Motorola. “We simply set out to answer the biological question of how the visual system works. And it turns out that this answer can be used to improve image processing by an order of magnitude.”
Prior to McCarthy’s research, it was generally assumed that the retina worked either like a simple camera or a biological compression system. This compression theory in particular stemmed from the fact that the retina contains 120-plus million photoreceptors, but only one million nerve fibers travel from the retina to the brain.
But McCarthy – at that time a Ph.D. student at the University of California, Berkeley – and W. Geoffrey Owen, Ph.D., Dean Emeritus of Biological Sciences and Professor Emeritus of Neurobiology and Neuroscience at the University of California, Berkeley, were convinced that the common thinking was wrong. They believed that instead of compressing signals, a big part of the retina’s function involved minimizing visual noise.
So more than a decade ago, McCarthy and Owen began studying the retina. And they made an important discovery about the way human beings see. They found that reducing uncertainty – not compressing signals – is the key to how the retina processes images.
“The retina codes uncertainty,” says McCarthy. “A lot of perceptual processing is simply figuring out the error – where an image is probably wrong.”
In very simplistic terms, McCarthy and Owen discovered that the retina is actually a sophisticated computational engine that “sees” by determining the error in its signal. Specifically, McCarthy and Owen found that when we look at objects, the retina “sees” something like the snow on an old TV screen. The retina’s first layer begins the process of translating that “snow” into a recognizable picture. It does this by focusing on patterns that search out the unexpected and minimize the noise.
Then the second layer of the retina determines how accurate the first layer was in reducing the noise. So what the retina sends the brain is not a “pixel map” of the visual world but instead of map of where, and by how much, its estimate of the visual world differs from its expectation of the world.
What the Retina and Video Processing Have in Common
Once McCarthy and Owen understood how the retina worked, they discovered that they could create an algorithm to mimic the retina’s function. They could then use this algorithm to reduce the “noise” in commercial video systems – without any unwanted side effects. Just like Dolby technology reduced the background noise found in audio without any side effects.
They named this revolutionary technology IPeG, short for the Integrated Perceptual Guide. And McCarthy and Owen used it to create the IPeG Engine, a patented video processing software that duplicates the process used by the human eye to format and convey visual information to the brain.
In fact, the IPeG engine actually transforms video signals into various neurologically based components. It then mimics the retina’s signal processing capabilities to reduce the amount of “noise” in these components before it transforms the signals back into the video realm. The result: better-quality images with less “noise,” which means that less bandwidth is needed to transport these images.
The IPeG technology has one big advantage compared to other compression technologies: It uses its knowledge of how the retina works to compress the parts of the image that are not considered important in visual processing. And that means that the compression does not introduce any unwanted side effects – which results in a much clearer image.
“If you are transmitting images and you are compressing them, you don’t want to compress the things that are important,” says Owen. “Instead, this technology manages the compression, so you don’t get a lot of the image processing artifacts that you get with other processing technologies.”
When these video signals are transformed back into the visual dimension by the IPeG engine, the result is a sharper, clearer image. The IPeG technology does not simply reduce the number of bits per pixels that need to be transported, which is what other compression technologies do. Instead, it transforms the way video is processed altogether.
Or as McCarthy and Owen put it: “IPeG takes image processing out of the realm of pixels and into the realm of perception.”
A Look at the Commercial Applications
The commercial applications for this technology are almost too numerous to count. But the bottom line is that the technology developed by McCarthy and Owen can improve compression efficiencies by 50 percent or more when it is used in conjunction with compression techniques already on the market today. And the resulting image is much higher quality than the images produced using today’s compression technologies.
For instance, satellite TV providers using today’s latest compression technologies can use IPeG compression firmware to increase the number of channels they can transmit across a transponder link by around 30 percent or more.
“That’s a game changer for them,” says McCarthy. “It lets them bypass the competition by delivering more HDTV channels and specialized sports services to the home.”
Providers of triple-play cable, data and telephone service – which are often very bandwidth-constrained because they are using the same pipe to deliver all three services – can use the IPeG technology to offer more high-definition channels to their customers. Or they can use the freed-up bandwidth to more than double the data speeds they deliver to their Internet users.
Using the new image processing technology, cable TV providers can double the number of channels they offer over a QAM, which stands for “quadrature amplitude modulation” and is the format used to encode and transmit digital cable channels.
The technology can also be used within digital set-top boxes to reduce the amount of noise in the stored images and thus reduce the amount of memory needed to store high-quality images. This means more hours of programming can be stored on less expensive set-top boxes. It also will actually sharpen the image delivered by these boxes.
The technology can also be used to monitor image quality and report that information back to the headend. And it can be used at the headend to transform content that was formatted for standard high-definition delivery into another format that might be more suitable for delivery on, for instance, a mobile phone.
Incorporating the technology into many of the video applications currently used by enterprise and government users would allow these applications to be supported at a much lower cost. For instance, using the technology, video security systems could store more recorded images using less memory – and could transfer the captured images across a much smaller bandwidth.
In addition, manufacturers can use this technology to create lower-cost video-enabled devices to support enterprise and government applications. For example, using the technology, manufacturers can develop black-and-white video screens that display much more recognizable images than today’s black-and-white screens. These screens could then be used to support applications such as digital signage, medical imaging, video security and traffic monitoring systems at a much lower cost.
Given that the technology is completely compatible with existing compression standards such as MPEG-2 and MPEG-4/H.264, it could also eventually become a key part of the evolution of future compression standards.
“This is a fundamental shift in how to optimize video and video delivery,” McCarthy says. “And the possibilities are just endless.”