The technology behind the line combines pattern recognition with augmented reality to deliver real-time game information for TV viewers.
First seen in 1998 on ESPN, the yellow line that marks a first down on the TV screen during football games has become a staple of NFL and NCAA college broadcasts. The technology has been used for more than a decade during high-stakes Superbowl games. Considered groundbreaking when it was introduced, the underlying technology that enables it was first invented in the 1990s at the David Sarnoff Research Center, now part of SRI.
The technology, Live Video Insertion System (L-VIS), uses high-resolution camera technology to provide accurate color and shape separation of small particles, capable of sorting even the smallest and thinnest particles, as small as 1mm.
During play, multiple cameras are used to pan across the field at all angles. Computer software and specialized algorithms combine the data alongside the 3D model, analyzing the mapped coordinates to show the line in the exact and appropriate positions based on gameplay. This shows the correct perspective for any given frame as the game proceeds.
Before each game, a team of experts uses augmented reality to create a 3D model of each specific playing field. This generates a map of colors at the time of play, including light conditions, field hues, uniform colors, etc. Color mapping is a very important part of the L-VIS that adds to the very specific shape and breaks needed to show a yellow line that is deeply integrated into the game.
Integrating augmented reality technology takes the objects that interfere with the yellow line into consideration, ensuring that each object isn’t covered by the line. The AR model makes it possible for the L-VIS to instantly change when an object, such as a player or a ball, gets in the way of the line.
The complex software removes the line over the object, or it would look like the line was painted on the object. The previously mapped colors in the system remove the appropriate parts of the yellow line and replace the appropriate colors giving the impression that the line is under any object in its way. The result is an integrated yellow line that seamlessly moves as the game is played.
The technology dates back to the early 1990s when a team at Sarnoff (which at the time operated independently of SRI) was hired to develop pattern recognition and tracking technology for various U.S. defense agencies. This work resulted in the vision-based technology that allows the virtual insertion of images and video into broadcast video signals live while the program is being televised.
In addition to the virtual yellow first-down marker, the technology has been used to place virtual advertising in broadcasts of soccer, baseball, and ice hockey games — and the pattern-recognition algorithms that work seamlessly on live-action television formed the foundational technology that would transform the TV experience for countless millions of sports fans.
