NASA’s Technology For Vision Algorithms To Catch Defects in Screen Displays

NASA has sent more than a few robotic missions into space, but it never loses sight of its goal to enable human exploration of the cosmos. A core component of planning for future manned missions is the Human Systems Integration Division, headquartered at Ames Research Center, which focuses on advancing our understanding of how people process information and interact with mechanical and electronic systems.

ProMetric I Imaging Colorimeter
In 2013 Radiant Zemax introduced the ProMetric I Imaging Colorimeter, designed especially for high-volume manufacturing customers, which utilizes the company’s NASA-improved TrueTest inspection software.

Andrew Watson is Ames’ senior scientist for vision research and the director of the Vision Group at the Human Systems Integration Division. “Many NASA missions in both space and aeronautics rely on vision,” he says, “either to monitor the environment around us or to gather information from visual displays. It is important to understand how the limits and capabilities of vision constrain and enable human performance in these missions.”

Part of Watson’s research focuses on optimizing vision technology, including the design of robotic vision systems. One of the long-standing problems in robotic vision is predicting the visibility of arbitrary targets, so Watson coordinated an international effort involving 10 research labs to collect data on it and developed a computer model that could predict those data.

“I also knew of many applied problems that would benefit from a simple model of target visibility,” he says, “so to serve that need, I converted the science model into an applied tool.”

Watson named his tool the Spatial Standard Observer (SSO), which has since been patented. The SSO is a simplified model of human vision, focusing on how contrast is perceived by the human eye. According to Watson, the SSO has potential applications in many areas, including vision for unmanned aerial vehicles, predicting outcomes for laser eye surgery, evaluating the legibility of text, and more.

Technology Transfer

One of the SSO’s applications soon caught the eye of Radiant Imaging—a company that would later merge with Zemax Development Corporation to become Redmond, Washington-based Radiant Zemax LLC. The company developed systems and software for testing light and lighting display systems and at the time was developing software tools to support its imaging colorimeter, a camera that measures color and brightness in accordance with the International Commission on Illumination’s standards for human visual performance.

A typical high-definition display, like that on a flat-panel television, contains approximately 2 million pixels, any one of which could be defective—what are termed “dead” or “stuck-on” pixels. Dead pixels and defects like them are relatively easy to catch when testing manufactured units, whether the inspector is a camera, sensor, or human being. But there is another class of defect called blemishes—or mura—which are areas of changing brightness or color that could be anywhere on the display and rather subtle. As mura have no necessary, fixed qualities, they tend to be difficult to predict or find.

Says Hubert Kostal, vice president of sales and marketing at Radiant Zemax, “As we researched this problem, we asked, ‘How can we find these floating defects, which can be anywhere in the display image and be any shape?’ If you know what you’re looking for, you can do a scan for it, but you get to a point where you’re not even sure whether it will be oval-shaped, square-shaped, or a bunch of dot defects clustered together.”

Portable computing devices

What Radiant Zemax needed was an objective way to classify the shape of luminance and color differences as human beings see them, in a way that could be incorporated into its image analysis software. The industry-standard method of grading differences in color is the use of a just noticeable difference (JND) scale, with one unit of JND representing the minimal change in color between two areas that is required before an average human observer could notice the difference between them. And, says Kostal, “as it turned out, one of the outputs from Watson’s approach allowed us to use JND to assign numerical values to the severity of blemishes.”

Satisfied that it had found the right tool for the job, the company licensed the technology from NASA, combined some of the SSO algorithms with the company’s own software, and created a new product called TrueMURA (Spinoff 2008).

Benefits

Manufacturers of television, laptop, tablet, and phone displays usually operate on razor-thin profit margins and face a dilemma when inspecting their product: On the one hand, an inspection process that doesn’t detect flawed units will result in returns, repairs, and unhappy customers unlikely to buy from the company again. On the other hand, imaging colorimeters can be more sensitive to flaws than the human eye, and manufacturers lose a lot of money when they throw away good product that could have satisfied consumers—even if the inspection flagged it as defective.

“That’s the conundrum,” says Kostal. “When is a mura meaningful to a human observer? How severe is it?” With its NASA-improved software, Radiant Zemax helps display manufacturers ensure that they are catching enough of the right kind of mura. “The most important thing is not what defects you can get a camera to detect,” says Kostal. “What’s really important is what customers see; it’s only a defect when a customer can see it.”

Home entertainment center
One challenge facing display manufacturers is that their customers spend so many hours looking at the screen. Barely noticeable blemishes that pass inspection can over time become major distractions to the consumer.

Since its debut several years ago, TrueMURA has been tightly integrated with Radiant Zemax’s software tools. In 2012, the company announced its TrueTest software system, which incorporated TrueMURA along with a suite of other quality and defect detection tests into one package designed to better automate display inspections. The technology powers the company’s newest line of high-performance imaging colorimeters, which are particularly suited for use by high-volume manufacturers of flat panel displays.

The speed, resolution, and accuracy of these newer imaging colorimeters is increasingly allowing the company’s system to take over inspection functions that have traditionally required human technicians, who can sometimes produce imprecise or inconsistent judgments. “All kinds of variables come into play with human inspection,” says Kostal. “If I’m giving you 1,000 displays to inspect, you might only have 5–10 seconds to look at each one. A defect that barely reaches the level of human perception might not be seen by an inspector, but a consumer watching hours of television and movies on that display is definitely going to notice it.” Kostal also points out that, when compared to the cost of using human inspectors, companies using Radiant Zemax’s product will end up saving money within 1 year.

Radiant Zemax’s products are now being used by a large number of companies producing common consumer electronics like smartphones, tablets, laptops, and televisions. According to the company, TrueTest is deployed on hundreds of production lines and is testing millions of flat-panel displays. “I can guarantee you,” says Kostal, “that if you go into a major electronics store, you will find products there that were either tested or designed using this capability.”

The company has been very pleased with its experience licensing NASA technology. “We do really cool things at Radiant Zemax, but we’re fundamentally an engineering company,” says Kostal. “We want to develop and provide solutions to our customers. The information that we were able to get from NASA enabled us to move quickly to the forefront of the industry in terms of analysis, methodology, and techniques.”

TrueMURA™ and TrueTest™ are trademarks and ProMetric® is a
registered trademark of Radiant Zemax LLC.