How to measure video quality perception

Update 2(01/06/2016): Fixed reference video bitrate unit from Kbps to KBps

Update 1(10/16/2016): Anne Aaron presented the VMAF at the Demuxed 2016.

When working with videos, you should be focusing all your efforts on best quality of streaming, less bandwidth usage, and low latency in order to deliver the best experience for the users.

This is not an easy task. You often need to test different bitrates, encoder parameters, fine tune your CDN and even try new codecs. You usually run a process of testing a combination of configurations and codecs and check the final renditions with your naked eyes. This process doesn’t scale, can’t we just trust computers to check that?

bit rate (bitrate): is a measure often used in digital video, usually it is assumed the rate of bits per seconds, it is one of the many terms used in video streaming.

same resolution, different bitrates.

codec: is an electronic circuit or software that compresses or decompresses digital content. (ex: H264 (AVC), VP9, AAC (HE-AAC), AV1 and etc)

We were about to start a new hack day session here at and since some of us learned how to measure the noise introduced when encoding and compressing images, we thought we could play with the stuff we learned by applying the methods to measure video quality.

We started by using the PSNR (peak signal-to-noise ratio) algorithm which can be defined in terms of the mean squared error (MSE) in decibel scale.

PSNR: is an engineering term for the ratio between the maximum possible power of a signal and the power of corrupting noise.

First, you calculate the MSE which is the average of the squares of the errors and then you normalize it to decibels.

For 3D signals (colored image), your MSE needs to sum all the means for each plane (ie: RGB, YUV and etc) and then divide by 3 (or 3 * MAX ^ 2).

To validate our idea, we downloaded videos (720p, h264) with the bitrate of 3400 kbps from distinct groups like News, Soap Opera and Sports. We called this group of videos the pivots or reference videos. After that, we generated some transrated versions of them with lower bitrates. We created 700 kbps, 900 kbps, 1300 kbps, 1900 kbps and 2800 kbps renditions for each reference video.

Heads Up! Typically the pivot video (most commonly referred to as reference video), uses a truly lossless compression, the bitrate for a YUV420p raw video should be 1280x720x1.5(given the YUV420 format)x24fps /1000 = 33177.6KBps, far more than what we used as reference (3400KBps).

We extracted 25 images for each video and calculate the PSNR comparing the pivot image with the modified ones. Finally, we calculate the mean. Just to help you understand the numbers below, a higher PSNR means that the image is more similar to the pivot.

700 kbps 900 kbps 1300 kbps 1900 kbps 2800 kbps 3400 kbps
Soap Op. 35.0124 36.5159 38.6041 40.3441 41.9447
News 28.6414 30.0076 32.6577 35.1601 37.0301
Sports 32.5675 34.5158 37.2104 39.4079 41.4540
A visual sample.

We defined a PSNR of 38 (from our observations) as the ideal but then we noticed that the News group didn’t meet the goal. When we plotted the News data in the graph we could see what happened.

The issue with the video from the News group is that they’re a combination of different sources: External traffic camera with poor resolution, talking heads in a studio camera with good resolution and quality, some scenes with computer graphics (like the weather report) and others. We suspected that the News average was affected by those outliers but this kind of video is part of our reality.

The different video sources are visible in clusters. (PSNR(frames))

We needed a better way to measure the quality perception so we searched for alternatives and we reached one of the Netflix’s posts: an approach toward a practical perceptual video quality metric (VMAF). At first, we learned that PSNR does not consistently reflect human perception and that Netflix is creating ways to approach this with the VMAF model.

They created a dataset with several videos including videos that are not part of the Netflix library and put real people to grade it. They called this score of DMOS. Now they could compare how each algorithm scores against DMOS.

FastSSIM, PSNRHVS, PSNR and SSIM (y) vs DMOS (x)

They realized that none of them were perfect even though they have some strength in certain situations. They adopted a machine-learning based model to design a metric that seeks to reflect human perception of video quality (a Support Vector Machine (SVM) regressor).

The Netflix approach is much wider than using PSNR alone. They take into account more features like motion, different resolutions and screens and they even allow you train the model with your own video dataset.

“We developed Video Multimethod Assessment Fusion, or VMAF, that predicts subjective quality by combining multiple elementary quality metrics. The basic rationale is that each elementary metric may have its own strengths and weaknesses with respect to the source content characteristics, type of artifacts, and degree of distortion. By ‘fusing’ elementary metrics into a final metric using a machine-learning algorithm – in our case, a Support Vector Machine (SVM) regressor”

Netflix about VMAF

The best news (pun intended) is that the VMAF is FOSS by Netflix and you can use it now. The following commands can be executed in the terminal. Basically, with Docker installed, it installs the VMAF, downloads a video, transcodes it (using docker image of FFmpeg) to generate a comparable video and finally checks the VMAF score.

You saved around 1.89 MB (37%) and still got the VMAF score 94.

Using a composed solution like VMAF or VQM-VFD proved to be better than using a single metric, there are still issues to be solved but I think it’s reasonable to use such algorithms plus A/B tests given the impractical scenario of hiring people to check video impairments.

A/B tests: For instance, you could use X% of your user base for Y days offering them the newest changes and see how much they would reject it.

Functional Programing 101 :: WWH


WWH: What? Why? How?

  1. What: a quick (hopefully, useful to real world) guide to functional programing using JavaScript strongly based on most adequate book.
  2. Why: it might empower you to write more robust programs: reusable, shorter, easier to reason about, less prone to error among others.
  3. How: by providing a quick textual introduction (WWH) followed by a simple code example and when possible a real code example.

Intro :: concepts

Functional Programing

What: a way to build code in which you use functions as the main design tool.

Why: might lead to code that’s easier to test, debug, parallelize, and understand.

How: thinking about what programs should do instead of how, using functions as the major unit to solve problems on computer.

First Class Functions

What: “functions are like any other data type and there is nothing particularly special about them – they may be stored in arrays, passed around, assigned to variables.”

Why: use functions to compose programs in a style that you can easily reason about, maintain, reuse and grow.

How: just create and use functions to solve problems.

Pure Functions

What: “a function that, given the same input, will always return the same output and does not have any observable side effect.”

Why: with pure functions we can easily cache, debug, test and parallelize the processing of them. There is no state to understand / set up.

How: write functions that does not have side effect. Although we’ll eventually write programs that mutate values, we can certainly try to minimize it. (And when we do need to mutate values, we can use functions to help us)

Basic toolbox :: currying

What: “You can call a function with fewer arguments than it expects. It returns a function that takes the remaining arguments.”

Why: you can promote the reusability to function level, you can use them to compose programs that expects another function

How: build a function with n parameters that returns n functions instead of the immediate result.

Medium toolbox :: composing

What: is the act of creating your programs using lots of functions.

Why: this promotes the reuse at a great level and forces you to think about what instead of how.

How: chain functions to produce a new callable function.

Example :: motivational

What: a better example to motivate you to go further with functional programing.

Why: most near real world examples are great to motivate you to learn something.

How: since you can see all the concepts together, I think you’ll notice the value.

You can see the example running at and check the commented code down bellow.

Screen Shot 2016-04-27 at 2.17.19 PM

Advanced toolbox & conclusion

I hope you might see the benefits you can have from using one or other technique from functional programming but for sure there are other benefits not shown here, I strongly recommend you to read the INCREDIBLE free book (gitbook) “Professor Frisby’s Mostly Adequate Guide to Functional Programming”, in fact, most of the ideas and examples here are from it.

There are advanced techniques to deal with data mutation with less pain, to handle errors and exceptions without try and catch and more abstractions that can help you and you can read them on the book.

And don’t use the handcrafted curry and compose built here (they’re far from production-ready), instead use a library like Ramda, which provides many basic functions like: map, filter and other all of them already curried, or lodash-fp.

Yeah, there no monado here. A special thank to Daniel Martins and Juarez Bochi, they helped a lot.


3 tips to make you a better developer



I’m sorry for the clickbait headline, I didn’t have a better idea/name for it.

We (developers) occasionally produce lazy/messy code and from time to time we need to remember the most important rule: “We do code to solve problems but also for human being be able: to use, to maintain and to evolute”. 

TLDR; (a unit can be a: function, var, method, class, parameter and etc)

  1. Naming your units with care and meaning;
  2. Try to see your code as a series of transformation;
  3. When possible make yours units generic.

Keep in mind that these tips are just my opinions and at the best they were based on: excellent books (Refactoring, DDD, Clean Coder and etc ), articles & blog posts,  excellent people I’ve worked/paired with,  presentations,  tweets and experiences.

naming is hard

Name your units with care and meaning. Your code should be easy to understand.

Although naming things is really hard, it is also extremely important. Let’s a see a snippet of code:

Let’s discuss about this code above:

  • the function topComments receives an id but is it the id from the comment, user, article? Let’s say it’s form the user, therefore userId should vanish this doubt.
  • the name of the function is topComments but it looks like it’s getting the top 10 latest comments only thus we could call it top10LatestCommentsFrom.
  • the ajax function accept two callbacks one in case of success (succCB) and otherwise an error (errCB), I believe we can call them: onSuccess and onError for better understanding.
  • all the arguments are using short names and we can have less confusing names just by using the entire name.
  • you got the ideia, naming things to let the code clear!

Although we still have so many problems in this code, now it’s easier to understand and we only named things properly.

For sure there are some cases when short names are just okay, for example: when you’re developing an emulator or virtual machine you often use short names like sp (stack pointer) and pc (program counter) or even doing a very generic unit.

filter -> union -> compact -> kick

Try to see and fit your code as transformations, one after another.

Some say that in computer science almost all the problems can be reduced to only two major problems: sort and count things (plus doing these in a distributed environment), anyway the point is: we usually are coding to make transformation over data.

For instance our function top10LatestCommentsFrom could be summarized in these steps:

  1. fetch comments (all)
  2. sort them (by date)
  3. filter them (only top)
  4. select the first 10

Which are just transformations over an initial list, we can make our function top10LatestCommentsFrom much better with that mindset.


By the way this could lead you to easily understand the new kid on the block sometimes referred as Functional Reactive Programming.

<be generic>

Work to make your units generic.

Let’s imagine you are in an interview process and your first task is to code a function which prints the numbers 1, 2 and 3 concatenated with “Hello, “. It should print: “Hello, 1” and then “Hello, 2″…

Now they ask you to print also the letters: “D”, “K” and “C”.

It was the first step toward the “generic”, now the interviewers say you have also to print a list of person’s name but now it’ll be a list of objects [{name: “person”},…].

Things start to get specific again and the interviewers want to test you. They ask you to print a list of car’s brand [{brand: “Ferrari”}, ..] plus a list of game consoles with their architecture [{name: “PS4”, arch: “x86-64”}, …]

Yikes, I suppose you’re not proud of that code and probably your interviewers will be little concerned about your skills with development, let’s list some of the problems with this approach.

  • Naming (we’re calling a person of an item)
  • High coupling (the function print knows too much about each printable)
  • Lots of (inner) conditionals 😦 it’s really hard to read/maintain/evolute this code

What we can do?! Well, it seems that all we need to do is to iterate through an array and prints an item but each item will require a different way of printing.


I said naming is important but when you make something very generic you should also make the abstract names not tied to any concrete concept. In fact, in Haskell (let’s pretend I know Haskell) when a concrete type of something may vary we use single letters to take their place.

Bonus round

  1. Make your units of execution to perform a single task.
  2. Use dispatch/pattern matching/protocol something instead of conditionals.
  3. Enforce DRY as much as you can.

presentation – Live Video Platform for FIFA World Cup

In this talk, we will describe’s live video stream architecture, which was used to broadcast events such as the FIFA World Cup (with peak of 500K concurrent users), Brazilian election debates (27 simultaneous streams) and BBB (10 cameras streaming 24/7 for 3 months) .

NGINX is one of the main components of our platform, as we use it for content distribution, caching, authentication, and dynamic content. Besides our architecture, we will also discuss the Nginx and Operational System tuning that was required for a 19Gbps throughput in each node, the open source Cassandra driver for Nginx that we developed, and our recent efforts to migrate to nginx-rtmp.

presentation QCon 2015 – ptBR


In this presentation you’ll see how we developed (what we used) the live video platform for the FIFA World Cup 2014. It shows how we made it scalable using lots of open source solutions.

Keywords: linux, cassandra, nginx, redis, BGP, logstash, graphite, python, ruby, lua

How To Optimize Nginx Configuration for HTTP/2 TLS (SSL)


http/2 over tls with nginx is already a reality, how can we achieve the best performance of it? check the example configuration.


We all know that http/2 is right here and although it doesn’t impose the TLS usage, the major browsers already took their side (a.k.a only supporting http/2 over TLS).

The support for http/2 was released with nginx 1.9.5 (except for “Server Push”). But isn’t HTTPS a lot slower than good old HTTP? Well, this is not easy to answer but we can fine tune nginx to do much better than the default configuration.

I really believe that the biggest fight is against latency not CPU load, the tips you’ll see here are mostly about reducing RTT in order to decrease latency.

tls ssl does not need cpu

Before we move on to the practical tips, let’s revise the simple tasks you must to do first:

  1. upgrade to the latest kernel (3.7+)
  2. upgrade to the latest openssl (1.0.1j)
  3. upgrade to the latest nginx (1.9.5)

These tips above will get you a lot of improvements but let’s go to the optimization tips:

TLS session resumption


When you want to use HTTPS, your browser needs to negotiate the session (certificate, cipher, hash algorithm, tls version, key …), in a very simplistic way it does follow the steps:

  1. Establish a TCP connection (SYN, SYN/ACK, ACK)
  2. Negotiate and establish the TLS session

When you leave the site and come back later, the browser will need to renegotiate the session. TLS session resumption is the technique to partially skip this negotiation by persisting the session for later usage.

The left graph represents an over simplified version of a full TLS handshake (skipping TCP handshake) and on the right side you can see how TLS resumption works, the point is to skip RTT.

tls negotiation  tls negotiation resumption


If we skip part of the session negotiation we’ll delivery fast content.


We do have two ways of solving this issue: saving the session (TLS) on server (session cache) or preferable on client (session ticket).

session cache

ssl_session_cache   shared:SSL:10m;
ssl_session_timeout 1h;

In this case when client tries to reconnect, the server will try to recovery past persisted session skipping partially the negotiation. With this shared session (of 10m), nginx will be able to handle 10 x 4000 sessions and the sessions will be valid for 1 hour.

However, there are problems with this approach:

  1. sessions are stored on the server;
  2. the “shared session” is saved in each server, so multiple nginx’s will not share the same session;

For the second problem, the great project openresty is about to release a new feature (ssl_session_store_by_lua) which will enable us to save these sessions in a “central” repository (like redis).

session ticket

# $> openssl rand 48 > file.key
ssl_session_tickets on; 
ssl_session_ticket_key file.key;

In this case, the server will create a ticket and send it to the client, when the client tries to connect again it’ll use the ticket and the server will just resume the session.

Nginx comes with session tickets enabled by default but if you will deploy your application in more than one box (bare metal, cloud, virtual machines, containers …) you’ll also need to specify the same key (used to create the tickets), you should rotate this key often.

Although this approach is much better than session cache, not all browsers support this so you might need to offer both solutions.

TLS false start


How about to have the same benefit (skipping RTT) as in TLS resumption but when the browser first negotiate with the server?! This is possible by using and enforcing the forward secrecy.

Instead of waiting the last handshake step from server, the browser will already send the data (request) and the server will reply with the data (response). This technique is known as TLS false start.

tls negotiation  tls false start


Less RTT means faster site/video/image/data to final users.


This is possible because we can extend the tls protocol by instrument it with specific ciphers.

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;

OCSP stapling and certificate chain


Creating trust is a hard task to do and part of the responsibilities of TLS is to enforce it. In order to establish this trust the browser (or you OS) needs to at least have one point to trust.

In a very simplistic way, your browser believes that is which it claims to be based on a chain of trust, it checks by:

  1. looking at its certificate and then checking if its signature is valid (checking all the certificates until the ROOT)
  2. looking if the certificate is not revoked by either searching it in CRL (certificate revocation list) or issuing a new request OCSP (Online Certificate Status Protocol)

tls chain

Both steps might force your browser to do more RTT, if your server doesn’t provide the intermediate certificates the browser will need to download them and it might even request an OSCP (requiring more RTT: DNS, TCP_HANDSHAKE, TLS_HANDSHAKE).


Less RTT means faster site/video/image/data to final users 2.


You can concatenate your certificate with its chain (except ROOT, it’s not necessary, in fact the browser won’t trust your ROOT) then avoiding extra RTT to download the missing certificates.

$ cat mysite.cert ca1.cert > full.cert
ssl_certificate /path/to/full.cert

You can set up nginx to avoid OCSP by stapling (this “staple” is digitally signed which makes possible for your browser to check its authenticity) the OCSP response on your server then you will avoid extra RTT for OCSP.

ssl_stapling on; 
ssl_stapling_verify on; 
ssl_trusted_certificate /path/to/cas.pem

TLS record size optimization


Although the maximum size of a TCP packet is 64K, a new TCP connection starts with much less than this maximum.

And each TLS record can hold at maximum 16K (which is the default size for nginx), summing up this size plus the headers of tcp and ip the server might need to make 2 RTT to serve the first bytes. And that’s not cool.

TCP is great but it has limitations, it is not ideal for all kinds of applications and there is even “quic” efforts to make web faster with experimentations using UDP instead of TCP.

Since we’ve reached our current speed limits, light speed, (who knows what “quantum” can do) we’re moving to avoid extra RTT.

*you can’t use QUIC on nginx yet.


Less RTT means faster site/video/image/data to final users. 🙂 Again!!!


There is a tradeoff here, you can either chose throughput (TLS record size to max) or latency (a small record size). It would be great if nginx could offer an adaptive option, starting small (4K, to speed up the first bytes) and after 1 minute or 1MB it increases for 16K.

ssl_buffer_size 16k; #for throughput, video applications
#ssl_buffer_size 4k; for quick first byte delivery

HSTS (HTTP Strict Transport Security)


HSTS “converts” your site to a strict HTTPS-only, it eliminates unnecessary HTTP-to-HTTPS redirects by shifting this responsibility to the client, most of the browsers support it. Even if you forgot to change http for https the browser will do that for you.


Redirects means more RTT, yeah I know it’s getting repetitive but it’s to reduce latency.


A simple http header to instruct the browser.

add_header Strict-Transport-Security "max-age=31536000; includeSubdomains"; 
#'max-age' values specifies how long browser should follow this rule.

Why Chrome doesn’t show/accept http2?

Users of the Google Chrome web browser are seeing some sites that they previously accessed over HTTP/2 falling back to HTTP/1. You can check what, why and how at nginx site.


It’s all about making web faster avoiding RTT (later I’ll post tips specific to http/2), so here’s a check list:

  • Upgrade to the latest: kernel, openssl and nginx.
  • Use TLS resumption and TLS false start
  • `cat` your certificate with the intermediates
  • Think about the best size (hard) for you TLS record
  • Enforce HSTS 😉

Here’s the full config example:

# command to generate dhparams.pen
# openssl dhparam -out /etc/nginx/conf.d/dhparams.pem 2048

limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m;
limit_req_zone $binary_remote_addr zone=req_limit_per_ip:10m rate=5r/s;
limit_req_status 444;
limit_conn_status 503;

proxy_cache_path /var/lib/nginx/proxy levels=1:2 keys_zone=backcache:8m max_size=50m;
proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args";
proxy_cache_valid 404 1m;

upstream app_server {
  server unix:/tmp/unicorn.myserver.sock fail_timeout=0;

server {
  listen 80;
  server_name *;
  limit_conn conn_limit_per_ip 10;
  limit_req zone=req_limit_per_ip burst=10 nodelay;
  return 301 https://$host$request_uri$is_args$args;

server {
  listen 443;
  server_name _;

  limit_conn conn_limit_per_ip 10;
  limit_req zone=req_limit_per_ip burst=10 nodelay;

  ssl on;

  ssl_stapling on;
  ssl_stapling_verify on;
  ssl_trusted_certificate /etc/nginx/conf.d/ca.pem;

  ssl_certificate /etc/nginx/conf.d/ssl-unified.crt;
  ssl_certificate_key /etc/nginx/conf.d/private.key;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_dhparam /etc/nginx/conf.d/dhparams.pem;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;
  ssl_session_timeout 10m;

  root /home/deployer/apps/;

  gzip_static on;
  gzip_http_version   1.1;
  gzip_proxied        expired no-cache no-store private auth;
  gzip_disable        "MSIE [1-6]\.";
  gzip_vary           on;

  client_body_buffer_size 8K;
  client_max_body_size 20m;
  client_body_timeout 10s;
  client_header_buffer_size 1k;
  large_client_header_buffers 2 16k;
  client_header_timeout 5s;

  add_header Strict-Transport-Security "max-age=31536000; includeSubdomains"; 

  keepalive_timeout 40;

  location ~ \.(aspx|php|jsp|cgi)$ {
    return 404;

  location ~* ^/assets/ {
    root /home/deployer/apps/;
    # Per RFC2616 - 1 year maximum expiry
    expires 1y;
    add_header Cache-Control public;
    access_log  off;
    log_not_found off;

    # Some browsers still send conditional-GET requests if there's a
    # Last-Modified header or an ETag header even if they haven't
    # reached the expiry date sent in the Expires header.
    add_header Last-Modified "";
    add_header ETag "";

  try_files $uri $uri/index.html $uri.html @app;

  location @app {
    proxy_set_header X-Url-Scheme $scheme;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # enable this if you forward HTTPS traffic to unicorn,
    # this helps Rack set the proper URL scheme for doing redirects:
    proxy_set_header X-Forwarded-For-Forwarded-Proto $https;

    proxy_set_header Host $host;
    proxy_redirect off;
    proxy_pass http://app_server;

  error_page 500 502 503 504 /500.html;
  location = /500.html {
    root /home/deployer/apps/;

With this configuration I was able to get an A+ at SSLlabs, this is a useful tool where you can check what you need to do to make your SSL site better, it gives you specific tips (apache, nginx, IIS).

tls a+
I strongly recommend you: