Watch the video.
Update 2(01/06/2016): Fixed reference video bitrate unit from Kbps to KBps
When working with videos, you should be focusing all your efforts on best quality of streaming, less bandwidth usage, and low latency in order to deliver the best experience for the users.
This is not an easy task. You often need to test different bitrates, encoder parameters, fine tune your CDN and even try new codecs. You usually run a process of testing a combination of configurations and codecs and check the final renditions with your naked eyes. This process doesn’t scale, can’t we just trust computers to check that?
bit rate (bitrate): is a measure often used in digital video, usually it is assumed the rate of bits per seconds, it is one of the many terms used in video streaming.
We were about to start a new hack day session here at Globo.com and since some of us learned how to measure the noise introduced when encoding and compressing images, we thought we could play with the stuff we learned by applying the methods to measure video quality.
PSNR: is an engineering term for the ratio between the maximum possible power of a signal and the power of corrupting noise.
First, you calculate the MSE which is the average of the squares of the errors and then you normalize it to decibels.
For 3D signals (colored image), your MSE needs to sum all the means for each plane (ie: RGB, YUV and etc) and then divide by 3 (or 3 * MAX ^ 2).
To validate our idea, we downloaded videos (720p, h264) with the bitrate of 3400 kbps from distinct groups like News, Soap Opera and Sports. We called this group of videos the pivots or reference videos. After that, we generated some transrated versions of them with lower bitrates. We created 700 kbps, 900 kbps, 1300 kbps, 1900 kbps and 2800 kbps renditions for each reference video.
Heads Up! Typically the pivot video (most commonly referred to as reference video), uses a truly lossless compression, the bitrate for a YUV420p raw video should be 1280x720x1.5(given the YUV420 format)x24fps /1000 = 33177.6KBps, far more than what we used as reference (3400KBps).
We extracted 25 images for each video and calculate the PSNR comparing the pivot image with the modified ones. Finally, we calculate the mean. Just to help you understand the numbers below, a higher PSNR means that the image is more similar to the pivot.
|700 kbps||900 kbps||1300 kbps||1900 kbps||2800 kbps||3400 kbps|
We defined a PSNR of 38 (from our observations) as the ideal but then we noticed that the News group didn’t meet the goal. When we plotted the News data in the graph we could see what happened.
The issue with the video from the News group is that they’re a combination of different sources: External traffic camera with poor resolution, talking heads in a studio camera with good resolution and quality, some scenes with computer graphics (like the weather report) and others. We suspected that the News average was affected by those outliers but this kind of video is part of our reality.
We needed a better way to measure the quality perception so we searched for alternatives and we reached one of the Netflix’s posts: an approach toward a practical perceptual video quality metric (VMAF). At first, we learned that PSNR does not consistently reflect human perception and that Netflix is creating ways to approach this with the VMAF model.
They created a dataset with several videos including videos that are not part of the Netflix library and put real people to grade it. They called this score of DMOS. Now they could compare how each algorithm scores against DMOS.
They realized that none of them were perfect even though they have some strength in certain situations. They adopted a machine-learning based model to design a metric that seeks to reflect human perception of video quality (a Support Vector Machine (SVM) regressor).
The Netflix approach is much wider than using PSNR alone. They take into account more features like motion, different resolutions and screens and they even allow you train the model with your own video dataset.
“We developed Video Multimethod Assessment Fusion, or VMAF, that predicts subjective quality by combining multiple elementary quality metrics. The basic rationale is that each elementary metric may have its own strengths and weaknesses with respect to the source content characteristics, type of artifacts, and degree of distortion. By ‘fusing’ elementary metrics into a final metric using a machine-learning algorithm – in our case, a Support Vector Machine (SVM) regressor”
The best news (pun intended) is that the VMAF is FOSS by Netflix and you can use it now. The following commands can be executed in the terminal. Basically, with Docker installed, it installs the VMAF, downloads a video, transcodes it (using docker image of FFmpeg) to generate a comparable video and finally checks the VMAF score.
You saved around 1.89 MB (37%) and still got the VMAF score 94.
Using a composed solution like VMAF or VQM-VFD proved to be better than using a single metric, there are still issues to be solved but I think it’s reasonable to use such algorithms plus A/B tests given the impractical scenario of hiring people to check video impairments.
A/B tests: For instance, you could use X% of your user base for Y days offering them the newest changes and see how much they would reject it.
Motivated by a friend, we’ll share bits of our experience during the Olympic Games Rio 2016. Before starting, I would like to clarify that Globo.com only had rights for streaming the content to Brazil.
We used around 5.5 TB of memory with 1056 CPU’s across two PoP’s located on the southeast of the country.
Not so long; I’ll read it
The live streaming infrastructure for the Olympics was an enhancement iteration over the previous architecture for FIFA 2014 World Cup.
The ingest point receives an RTMP input using nginx-rtmp and then forwards the RTMP to the segmenter. This extra layer provides mostly scheduling, resource sharing and security.
Now let’s move to the user point of view. When the player wants to play a video, it needs to get a video chunk, requesting a file from our front-end, which provides caching, security, load balancing using nginx.
Modern network cards offers multiple-queues: pin each queue, XPS, RPS to a specific cpu.
When this front-end does not have the requested chunk it goes to the backend which uses nginx with lua to generate the playlist and serve the video chunks from cassandra.
This is just a macro view, for sure we also had to provide and scale many micro services to offer things like live thumb, electronic program guide, better usage of the ISP bandwidth, geofencing and others. We deployed them either on bare metal or tsuru.
In the near future we might investigate other adaptive stream format like dash, explore other kinds of input (not only RTMP), increase the number of bitrates, promote a better usage of our farm and distribute the content near of the final user.
Thanks @paulasmuth for pointing out some errors.
WWH: What? Why? How?
- Why: it might empower you to write more robust programs: reusable, shorter, easier to reason about, less prone to error among others.
- How: by providing a quick textual introduction (WWH) followed by a simple code example and when possible a real code example.
Intro :: concepts
What: a way to build code in which you use functions as the main design tool.
Why: might lead to code that’s easier to test, debug, parallelize, and understand.
How: thinking about what programs should do instead of how, using functions as the major unit to solve problems on computer.
First Class Functions
What: “functions are like any other data type and there is nothing particularly special about them – they may be stored in arrays, passed around, assigned to variables.”
Why: use functions to compose programs in a style that you can easily reason about, maintain, reuse and grow.
How: just create and use functions to solve problems.
What: “a function that, given the same input, will always return the same output and does not have any observable side effect.”
Why: with pure functions we can easily cache, debug, test and parallelize the processing of them. There is no state to understand / set up.
How: write functions that does not have side effect. Although we’ll eventually write programs that mutate values, we can certainly try to minimize it. (And when we do need to mutate values, we can use functions to help us)
Basic toolbox :: currying
What: “You can call a function with fewer arguments than it expects. It returns a function that takes the remaining arguments.”
Why: you can promote the reusability to function level, you can use them to compose programs that expects another function
How: build a function with n parameters that returns n functions instead of the immediate result.
Medium toolbox :: composing
What: is the act of creating your programs using lots of functions.
Why: this promotes the reuse at a great level and forces you to think about what instead of how.
How: chain functions to produce a new callable function.
Example :: motivational
What: a better example to motivate you to go further with functional programing.
Why: most near real world examples are great to motivate you to learn something.
How: since you can see all the concepts together, I think you’ll notice the value.
You can see the example running at https://jsfiddle.net/swmrmgur/2/ and check the commented code down bellow.
Advanced toolbox & conclusion
I hope you might see the benefits you can have from using one or other technique from functional programming but for sure there are other benefits not shown here, I strongly recommend you to read the INCREDIBLE free book (gitbook) “Professor Frisby’s Mostly Adequate Guide to Functional Programming”, in fact, most of the ideas and examples here are from it.
There are advanced techniques to deal with data mutation with less pain, to handle errors and exceptions without try and catch and more abstractions that can help you and you can read them on the book.
And don’t use the handcrafted curry and compose built here (they’re far from production-ready), instead use a library like Ramda, which provides many basic functions like: map, filter and other all of them already curried, or lodash-fp.