Content adaptive encoding (CAE) is gaining ground rapidly by enabling a substantial reduction in bit-rate in addition to the standard codec for a given video quality. This comes at a time when growth in online distribution is putting pressure on network bandwidth. There are now at least ten vendors in the field, while support for CAE is increasingly taken into account when awarding contracts for OTT platforms.
The trend is being driven not just by the stampede towards online video distribution but also emergence of services that include Ultra HD (UHD) content, which consumes a lot more bandwidth, especially when higher frame rates are incorporated as well as 4K resolutions at 2160 x 3840 pixels. That is why the Ultra HD Forum, responsible for orchestrating standards for UHD infrastructure as opposed to the cameras and viewing devices at either end, has added CAE to its list of core technologies for its Phase B second wave specifications.
The Ultra HD Forum is most interested in applying CAE to live content because that is where bandwidth pressure will be felt most as linear OTT services expand. At present however, use of CAE is largely confined to on-demand content because it is computationally intensive, and without significant parallel processing its execution takes too long for live services.
Given that CAE takes advantage of variations in content complexity to reduce bandwidth for a given perceptual quality, by definition it varies significantly in its effectiveness. For sequences such as talking heads or animations, where there is little movement or where contrast is low (or both), bit rate can be reduced by up to 50% in addition to that achieved by conventional codecs such as HEVC or H.264, or even 75% in extreme cases.
On the other hand, for fast-moving sporting action with lots of contrast the limit for further reduction by CAE is only about 10%, or at most 15%. These figures are in line with tests by various vendors even if they all claim their implementation is superior to their competitors. Apart from differences in technology, some options, such as EuclidiQ’s Rithm, are hosted in Amazon Web Services, which makes integration into increasingly common cloud-based workflows easier.
Other significant vendors of CAE include Brightcove, Bitmovin, Harmonic, Capella Systems and Beamr. However, it was a service provider, Netflix, that first brought CAE into the limelight with its project to re-encode its entire catalogue through CAE to save bandwidth, something that took over two years and was completed early this year.
Previously Netflix had relied, like all other operators, on key ladders that equate each level of content quality to a given bit-rate. For example, 5.8 Mbps was deemed the minimum rate capable of sustaining 1080p ‘full HD’ while at the other extreme 235 Kbps was the minimum for 320×240 resolutions adequate only for small mobile phone screens. But Netflix, like others, had observed that at 5.8 Mbps some high action content would still exhibit some blockiness, while say a cartoon might be displayed without any such artefacts at full HD in just 1.7 Mbps, only about 30% as much.
For cartoons, Netflix’s re-encoding yielded the full potential of CAE with huge bit rate savings because there is little variation within the content. But this application of CAE on a per-title basis failed to take advantage of variations in quality within content, which can be considerable where fast action is involved. After all, during, say a football match, for some of the time there is little movement and that can only be catered for by applying CAE down to the scene or even frame level. For that reason, Netflix is considering repeating the process all over again.
Few service providers have Netflix’s resources to encode their own catalogue but now there is widespread demand for such adaptation to be performed as part of service delivery, according to Mark Donnigan, VP Marketing at Israeli CAE vendor Beamr. Donnigan claims that Beamr’s Content-Adaptive Bitrate rate-control can achieve additional bit-rate savings of 20-40% and recently reduced a UHD 4K HDR rec2020 file by 36% from 25Mbps to 16Mbps.
He also noted pressure coming from the consumer side. “Consumers now possess amazingly capable displays, whether it is the primary UHD TV in their house, or one of the new mobile phones from Apple or Samsung, that is capable of displaying 4K resolution video,” says Donnigan. “Even as video advancements are opening up new entertainment experiences, such as HDR and 4K, there will always be a need to reduce bit-rate so streaming services can deliver this content to the most users possible.”
The terms ‘context aware encoding’ and ‘content adaptive bit-rate control’ are sometimes used to describe the same approach. Donnigan claims Beamr originated the technology in 2009, when it settled on ‘content-adaptive’ to describe its perceptual optimisation technology.