Motivated by a friend, we’ll share bits of our experience during the Olympic Games Rio 2016. Before starting, I would like to clarify that Globo.com only had rights for streaming the content to Brazil.
It reached a peak of more than 400K simultaneous users, consuming a sum of more than 400000 Tb (400 Pb) of live stream video (peak around 600 Gbps), with an average bitrate of 2.0 Mbps. In our metric stack that gets data from the player, we got 225M messages (96GB) in one day.
All these numbers were achieved with a cluster with 5.5 TB of memory and 1056 CPU’s across two PoP’s located on the southeast of the country.
Not so long; I’ll read it
The live streaming infrastructure for the Olympics was an enhancement iteration over the previous architecture for FIFA 2014 World Cup.
The ingest point receives an RTMP input using nginx-rtmp and then forwards the RTMP to the segmenter. This extra layer provides mostly scheduling, resource sharing and security.
Now let’s move to the user point of view. When the player wants to play a video, it needs to get a video chunk, requesting a file from our front-end, which provides caching, security, load balancing using nginx.
Modern network cards offers multiple-queues: pin each queue, XPS, RPS to a specific cpu.
When this front-end does not have the requested chunk it goes to the backend which uses nginx with lua to generate the playlist and serve the video chunks from cassandra.
This is just a macro view, for sure we also had to provide and scale many micro services to offer things like live thumb, electronic program guide, better usage of the ISP bandwidth, geofencing and others. We deployed them either on bare metal or tsuru.
In the near future we might investigate other adaptive stream format like dash, explore other kinds of input (not only RTMP), increase the number of bitrates, promote a better usage of our farm and distribute the content near of the final user.
Thanks @paulasmuth for pointing out some errors.