Learn Good Design From Linux C Source Code

Learn how Linux/FFmpeg C partial codebase is organized to be extensible and act as if it were meant to have “polymorphism”. Specifically, we’re going to briefly explore how Linux concept of everything is a file works at the source code level as well as how FFmpeg can add support fast and easy for new formats and codecs.

diagram_components

 

Good software design – Introduction

To write useful and long term maintainable software we tend to look out for patterns and group them into abstractions and it seems that’s the case for devs behind Linux and FFmpeg too.

Software design

When we’re creating software, we’re building data structures and defining their behaviors and dependencies. The way we create and link them can be seen as the design/architecture of the software.

Let’s say we’re building a media framework that encodes/decodes video and audio. The codecs AV1, H264, HEVC, and AAC all do some common operations and if we can provide a generic abstraction that holds these common operations and data we can use this concept instead of relying on the concrete idea of what a specific codec does.

Through the years many developers noticed that software with a good design is a good idea that pays off as software grows in complexity.

This is one of the ideas behind the good design for software, to rely on components that are weakly linked and with boundaries around what it should do.

Maybe it’s easier to see all these concepts in practice. Let’s code a quick pseudo media stream framework that provides encoding and decoding for several codecs.

This pseudo-code in ruby tries to recreate what we’re discussing above, there is an implicit concept here of what operations a codec must have, in this case, the operations are encode and decode. Since ruby is a dynamically typed language any class can present these two operations and act as a codec for us.

Developers sometimes may use the words: contract, API, interface, behavior and operations as synonyms.

This design might be considered good because if we want to add a new codec we just need to provide an implementation and add it to the list, even the list could be built in a dynamic way but the idea is that this code seems easy to extend and maintain because it tries to keep link between the components weak (low coupling) and each component does only what it should do (cohese).

When we go (no pun intended) to a statically typed language like golang we need to be more formal, describing the required types but it’s still doable.

The interface type in golang is much more powerful than Java’s similar construct because its definition is totally disconnected from the implementation and vice versa.

In the C language we still can create the same behavior but it’s a little bit different.

Code inspired by https://www.bottomupcs.com/abstration.xhtml

We first define the abstract operations (functions in this case) in a generic struct and then we fill it with the concrete code, like the av1 decoder and encoder real code.

Many other languages have somewhat similar mechanisms to dispatch methods or functions as if they were part of an agreed protocol and then the system integration code can deal only with this high-level abstractions.

Linux Kernel – Everything is a file

Have you ever heard the expression everything is a file in Linux? The idea is to have a common interface for all kinds of resources in Linux, for instance, Linux handles network socket, special files (like /proc/cpuinfo) or even USB devices as files.

This is a powerful idea that can make easy to write programs for linux since we can rely in a set of well known operations. Let’s see this in action:

The struct file_operations define what one should expect from a concept of what file can do.

Here we can see the directory implementation of these operations for the ext4 file system.

And even the cpuinfo proc files rely on this abstraction.

FFmpeg – Formats

For input format files, FFmpeg creates a struct called AVInputFormat that is implemented by any format that wants to be used as input. MKV files fill this structure with its implementation as the MP4 format too.

Here’s an overview of FFmpeg flow/architecture that shows that the internal componets are linked mostly to the abstract concepts like AVCodec but not directly to their implemenation.

remuxing_libav_components
FFmpeg architecture view from transmuxing flow

 

This design allows new codecs, formats, and protocols to be integrated and released easier. DAV1d (an av1 open-source implementation) was integrated into FFmpeg May this year and you can follow along the commit diff to see how easy it was. In the end, it needs to register itself as an available codec and follow the expected operations.

No matter the language we use we can (or at least try to) build a software with low coupling and high cohesion in mind, these two basic properties can allow you to build easier to maintain and extend software.

Advertisements

presentation – Live Video Platform for FIFA World Cup


In this talk, we will describe globo.com’s live video stream architecture, which was used to broadcast events such as the FIFA World Cup (with peak of 500K concurrent users), Brazilian election debates (27 simultaneous streams) and BBB (10 cameras streaming 24/7 for 3 months) .

NGINX is one of the main components of our platform, as we use it for content distribution, caching, authentication, and dynamic content. Besides our architecture, we will also discuss the Nginx and Operational System tuning that was required for a 19Gbps throughput in each node, the open source Cassandra driver for Nginx that we developed, and our recent efforts to migrate to nginx-rtmp.

presentation QCon 2015 – ptBR

http://www.infoq.com/br/presentations/contos-de-escalabilidade-na-plataforma-de-videos-online-da-globo

qcon-rio-2015-escalabilidade-na-plataforma-de-vdeos-ao-vivo-29-1024

In this presentation you’ll see how we developed (what we used) the live video platform for the FIFA World Cup 2014. It shows how we made it scalable using lots of open source solutions.

Keywords: linux, cassandra, nginx, redis, BGP, logstash, graphite, python, ruby, lua

Will we only create and use dynamic languages in the future?

Since I’ve playing with some dynamic languages (Ruby and Clojure), I have been thinking about why would anybody create a new static typed language?! And I didn’t get the answer.

I started programming in Visual Basic and I taste its roots, which are almost all full of procedure commands (bunch of do, goto and end), then I moved to C#, sharper it changes the end’s for }’s and give us a little more power based on some premises: we can treat two different things in the same way, polymorphism. The last static language, but not the least, I used (and I use it) Java, abusing of his new way of treating a set of things equality, the interfaces and using its “powers” on reflections.

Although when I started to use Ruby I saw that I could treat a group of things equality without doing any extra work. I still need to code models and composed types, even though we can create or change them dynamically using “real power” of metaprogramming.

When I start to study and apply the Clojure and its principles, my first reaction was the rejection, how can I go on without my formal objects, how can I design software without a model in the head and so on. I wasn’t thinking about how actually I do software, currently I use TDD to design software and I don’t think what models I need to have, I do think in terms of “what I want”. At minimum, Clojure make me think about, do we really need object to design software?! .  A three days ago I saw an amazing video about similar thoughts: Some thoughts on Ruby after 18 months of Clojure.

Summarising: With my limited knowledge of theses languages, let’s suppose we use a function (which we don’t have source code) and we want to do something before that function is executed (intercept) using: VB I’ll need to check every single piece of code which we call this function and call another one, in Java we can use a AOP framework, in Ruby we can use the spells of metaprogramming. It seems that some frameworks, patterns and extra work aren’t needed more because of this dynamic language evolution.

My conclusions using dynamic languages (Clojure/Ruby) for now it’s: I write less code and reuse them more easy, so I don’t see any reason to create/use a new static typed language, would you see any motivation to do that?

PS: When I use C# (.Net Framework 1.3 – 2.0) it was not so super cool as today.

Functional programming with Clojure

Clojure

I’ve been studying the new language called Clojure (all the cool kids are talking about Clojure). It is a functional language created by Rich Hickey around 2007. This is a(nother) dialect of Lisp. It is a dynamic language as Ruby, JavaScript and others. As said before Clojure (pronounced as closure) it’s a impure functional language in contrast with Haskell, a pure functional language. It runs over the JVM, so it’s fast, interoperable with Java among a lots of good stuffs that JVM give us. To put hands-on and try code something you can use the try Clojure online or you can download the clojure.jar file and run it. Surprisingly Clojure it’s easy to learn.


java -jar clojure-x.x.x.jar

What it a functional language? (concepts)

first-order functions -> functions are treated as values. You can store a function on a variable, you can pass one function to another or you can return a function from another function.

var sum = function(a,b){
  return a + b;
};

var obj = function(sum){
  return {
    hello: "hello",
    sum: sum
  };
}();

obj.sum(3,5);

functions constructs -> the language constructs are function instead of keyword. Constructions for conditions (if), for iterations (for, while), catch exceptions (try, catch) and others.


(if condition do-it else-do-it)

stateless -> it’s functional in the sense of math, you have functions which defines values input and output and doesn’t rely on outside global state. In such pure function you won’t produce any side-effect (read, write outside resource). Obviously we will produce programs which causes side-effects, clojure helps you build “mutable” data . On other pure languages like Haskell side-effects are treated as expections so you have concepts like actors and monad.

immutable data -> collections and local variable, in clojure, are immutable. The immutability, helps us in parallelism, since the “values” are immutable you can shared then without worry about locks.

currying -> is the technique of transforming a function that takes multiple arguments (or an n-tuple of arguments) in such a way that it can be called as a chain of functions each with a single argument (partial application).

memoization -> is an optimization technique used primarily to speed up computer programs by having function calls avoid repeating the calculation of results for previously processed inputs.

Resources

RSpec and Watir to test web applications

Testing is cool

Software testing

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation. Test techniques include, but are not limited to, the process of executing a program or application with the intent of finding software bugs (errors or other defects) – Wikipedia

The main intend of this post is, introduce you to UI tests over some ruby toys. In fact you could create an entire project (new) in ruby just to test your legacy web project. It’s cool, you can learn new language and work for the improvement of your legacy product. If you are totally new for ruby maybe a ruby overview can help you. (or might confuse you more)

Installing ruby, watir and rspec

Instead of installing the ruby directly, we are going to install the RVM (Ruby Version Manager) to then install any ruby we need. The steps described here were made on Ubuntu 11.04. On your terminal do the magic to install RVM.

bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
echo ‘[[ -s “$HOME/.rvm/scripts/rvm” ]] && . “$HOME/.rvm/scripts/rvm” # Load RVM function’ >> ~/.bash_profile
source .bash_profile

And from now on, your life will be better on ruby interpreters versions. Let’s install the ruby 1.9.2. (terminal again)

rvm install 1.9.2

And if we want to see the rubies installed on our machine?

rvm list

And now, how can we chose one ruby to work on the terminal session?

rvm use 1.9.2

For the test purpose we will use Watir and RSpec, great tools for testing, make fun with BDD and the best thing is install them it’s very easy.

gem install watir-webdriver
gem install rspec

Hands-on

Since we have all things installed, we can move for the example. The feature I want to test is the search system of  Amazon. Being more precise, I want to search for ‘Brazil’ and see if the ‘Brazil on the Rise’ is within the results as I want to be sure when I search for ‘semnocao‘ the Amazon doesn’t provide any result. Now, we can write the spec.

require 'amazon_page'

describe AmazonPage do
 before(:each) do
   @page = AmazonPage.new
 end
 after(:each) do
   @page.close
 end
 it "should show 'Brazil on the Rise' when I query for [Brazil]" do
  @page.query 'Brazil'
  @page.has_text('Brazil on the Rise').should == true
 end
 it "should bring no result when I search for [semnocao]" do
  @page.query 'semnocao'
  @page.results_count.should == 0
 end
end

The specification is very simple, it will create a page before each test calling and close the page after each test calling. There is only two tests: test when you search for Brazil  and  when you search for semnocao. We will design the tests using page object pattern. The class bellow is the page which represents the Amazon page and all testable behaviors should be inside of it.

require 'watir-webdriver'

class AmazonPage
 def initialize
  @page = Watir::Browser::new :firefox
  @page.goto 'http://www.amazon.com'
 end
 def close
  @page.quit
 end
 def query(parameter)
  @page.text_field(:id=>'twotabsearchtextbox').set parameter
  @page.send_keys :enter
 end
 def has_text(text)
  @page.text.include? text
 end
 def results_count
    if @page.text.include? 'did not match'
     0
    else
     @page.div(:id=>'resultCount').text.split(' ')[5].gsub(',','').to_i
    end
 end
end

To run this you just need to type on your terminal.

rspec spec/

The final code can be downloaded or viewed at github.

Additions

  • We could improve our story readibility  with Cucumber.
  • We could send the browser execution to an Xvfb server. (A.K.A. running headless) The browser pops up really bothers me.
  • We could integrate it with our CI.
  • We could design a base Page class for provide common operations as mixin or something

ps: the post was very inspired by KK post and Saush.

Ruby overview

Introduction

Take a look at this, there is two classes: a person and a teacher. The person (originally) just know how to speak English and then teacher teach him speak other language, it can show you how powerful and beautiful ruby is.

class Person
 attr_accessor :name
 def speak_english
  puts "Hi people!"
 end
end

class BrPortugueseTeacher
 def teach(person)
  def person.speak_portuguese
   puts "Ola pessoal!"
  end
 end
end

bill_gates = Person.new
bill_gates.nome = "Bill Gates"
pasquale = Teacher.new
pasquale.teach bill_gates
#now bill gates knows portuguese
bill_gates.speak_portuguese

The intent here is just show a quick overview of ruby from a newbie.

Code’s comment

#one line comment
=begin
Multiply lines comment.
Given that ...
=end

String

String in ruby is mutable (but when you use the operator method + it creates another string, so to concatenate strings you should use the operator <<) and just a little tip avoid the concatenation by using + instead prefer use interpolation, a way to handle string very similar to expression language, and it’s faster than normal concatenation.

ran = 34434
who = "Leandro Moreira"
puts "#{who} generates this #{ran} number"

Conventions

Yet on mutability, when you write a method that can affect the internal state, you should use the bang operator (!) on method’s name.

old_source_name = "angeline"
puts old_source_name.capitalize
puts old_source_name.capitalize!

Another cool convention to Boolean methods is end them with ?

if account.cancelled?
 puts "Run Forest, run!"
end

Range object

In ruby you can use a type Range to describe ranges and its use is very easy.

zero_to_ten = (0..10) #inclusive
one_to_seven = (0...8) #exclusive
alphabetic = ('a'..'z') #you also can omit the (

It’s all object and quick tips

– Hey, language prints I win three times.

puts "I win " * 3

You can use anything on if statement and it can ben true or false (and nil which is false too).

A weird thing is one way of handle the regular expressions.

/myexp/ =~ "sentence"
#"sentence" matches myexp?

Another weird operator is or equals.

list ||= flights
#the list will just receive the flights if list is nil.

The classes are really open

One of the main features of ruby is Open Class, this is cool, you just can grab a class and write a new feature for it.

class String
 def do_nothing
  puts "doing nothig"
 end
end

And then you just call it.

"number".do_nothing

Let’s trick the addition operations on number.

class Fixnum
 def +(other)
  self - other
 end
end
puts 2+1
#and it will prints 1. (~:

Variable arguments

Sometimes you need to use this kind of flexibility.

user.buy computer
user.buy computer, mouse, monitor

To achieve this you just use the syntax. The splat operator how it is known.

def buy(*products)
 #buy logic
end

Hash enhancements

There is a lot of people which claims to use hash as parameter.

e_account.transfer :to_account => my, :value => 4800

def transfer (parameters)
 dest_account = parameters[:to_account]
 #...
end

Declarations

class Anything
 @field #object field
 @@field #class field
end

Singleton in Ruby

Singleton pattern is a design pattern used to implement the mathematical concept of a singleton, by restricting the instantiation of a class to one object. This is useful when exactly one object is needed to coordinate actions across the system. The concept is sometimes generalized to systems that operate more efficiently when only one object exists, or that restrict the instantiation to a certain number of objects (say, five). Some consider it an anti-pattern, judging that it is overused, introduces unnecessary limitations in situations where a sole instance of a class is not actually required, and introduces global state into an application.  (From wikipedia)

class HyperDao
@@instance = HyperDao.new
def self.instance
return @@instance
end
private_class_method :new
end

But we’re talking about ruby, don’t we?

require 'singleton'
class God
include Singleton
end

It’s done! 😀

Equals

If you want or need to rewrite the equals…

def ==(other)
 self.id = other.id
end

Duck typing – good-bye interface

Duck typing is a style of dynamic typing in which an object’s current set of methods and properties determines the valid semantics, rather than its inheritance from a particular class or implementation of a specific interface. The name of the concept refers to the duck test, attributed to James Whitcomb Riley (see History below), which may be phrased as follows:”When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.” (From Wikipedia)

class PremiumAccount
 def saldo
   #
 end
end

class CommonAccount
 def saldo
  #
 end
end

The bank manager will accept that.

class BankManager
 def total_debt(accounts)
  for account in accounts do
   debt += account.saldo
  end
 end
end

Mixin

Mixin is a class that provides a certain functionality to be inherited or just reused by a subclass, while not meant for instantiation (the generation of objects of that class). Inheriting from a mixin is not a form of specialization but is rather a means of collecting functionality. A class may inherit most or all of its functionality from one or more mixins through multiple inheritance. (Again, from Wikipedia)

module Logging
 def log(message)
  puts message
 end
end

class Anything
 include Logging
  #...
end

any = Anything.new
any.log "It started now!"

Metaprogramming

Metaprogramming is the writing of computer programs that write or manipulate other programs (or themselves) as their data, or that do part of the work at compile time that would otherwise be done at runtime. In many cases, this allows programmers to get more done in the same amount of time as they would take to write all the code manually, or it gives programs greater flexibility to efficiently handle new situations without recompilation.

class Person
 attr_accessor :name
 def speak_english
  puts "Hi people!"
 end
end

class BrPortugueseTeacher
 def teach(person)
  def person.speak_portuguese
   puts "Ola pessoal!"
  end
 end
end

bill_gates = Person.new
bill_gates.nome = "Bill Gates"
pasquale = Teacher.new
pasquale.teach bill_gates
#now bill gates knows portuguese
bill_gates.speak_portuguese

Highly influenced by ruby on rails from Caelum.