3 tips to make you a better developer

four20puzzle

introduction

I’m sorry for the clickbait headline, I didn’t have a better idea/name for it.

We (developers) occasionally produce lazy/messy code and from time to time we need to remember the most important rule: “We do code to solve problems but also for human being be able: to use, to maintain and to evolute”. 

TLDR; (a unit can be a: function, var, method, class, parameter and etc)

  1. Naming your units with care and meaning;
  2. Try to see your code as a series of transformation;
  3. When possible make yours units generic.

Keep in mind that these tips are just my opinions and at the best they were based on: excellent books (Refactoring, DDD, Clean Coder and etc ), articles & blog posts,  excellent people I’ve worked/paired with,  presentations,  tweets and experiences.

naming is hard

Name your units with care and meaning. Your code should be easy to understand.

Although naming things is really hard, it is also extremely important. Let’s a see a snippet of code:

Let’s discuss about this code above:

  • the function topComments receives an id but is it the id from the comment, user, article? Let’s say it’s form the user, therefore userId should vanish this doubt.
  • the name of the function is topComments but it looks like it’s getting the top 10 latest comments only thus we could call it top10LatestCommentsFrom.
  • the ajax function accept two callbacks one in case of success (succCB) and otherwise an error (errCB), I believe we can call them: onSuccess and onError for better understanding.
  • all the arguments are using short names and we can have less confusing names just by using the entire name.
  • you got the ideia, naming things to let the code clear!

Although we still have so many problems in this code, now it’s easier to understand and we only named things properly.

For sure there are some cases when short names are just okay, for example: when you’re developing an emulator or virtual machine you often use short names like sp (stack pointer) and pc (program counter) or even doing a very generic unit.

filter -> union -> compact -> kick

Try to see and fit your code as transformations, one after another.

Some say that in computer science almost all the problems can be reduced to only two major problems: sort and count things (plus doing these in a distributed environment), anyway the point is: we usually are coding to make transformation over data.

For instance our function top10LatestCommentsFrom could be summarized in these steps:

  1. fetch comments (all)
  2. sort them (by date)
  3. filter them (only top)
  4. select the first 10

Which are just transformations over an initial list, we can make our function top10LatestCommentsFrom much better with that mindset.

 

By the way this could lead you to easily understand the new kid on the block sometimes referred as Functional Reactive Programming.

<be generic>

Work to make your units generic.

Let’s imagine you are in an interview process and your first task is to code a function which prints the numbers 1, 2 and 3 concatenated with “Hello, “. It should print: “Hello, 1” and then “Hello, 2″…

Now they ask you to print also the letters: “D”, “K” and “C”.

It was the first step toward the “generic”, now the interviewers say you have also to print a list of person’s name but now it’ll be a list of objects [{name: “person”},…].

Things start to get specific again and the interviewers want to test you. They ask you to print a list of car’s brand [{brand: “Ferrari”}, ..] plus a list of game consoles with their architecture [{name: “PS4”, arch: “x86-64”}, …]

Yikes, I suppose you’re not proud of that code and probably your interviewers will be little concerned about your skills with development, let’s list some of the problems with this approach.

  • Naming (we’re calling a person of an item)
  • High coupling (the function print knows too much about each printable)
  • Lots of (inner) conditionals 😦 it’s really hard to read/maintain/evolute this code

What we can do?! Well, it seems that all we need to do is to iterate through an array and prints an item but each item will require a different way of printing.

 

I said naming is important but when you make something very generic you should also make the abstract names not tied to any concrete concept. In fact, in Haskell (let’s pretend I know Haskell) when a concrete type of something may vary we use single letters to take their place.

Bonus round

  1. Make your units of execution to perform a single task.
  2. Use dispatch/pattern matching/protocol something instead of conditionals.
  3. Enforce DRY as much as you can.
Advertisements

presentation – Live Video Platform for FIFA World Cup


In this talk, we will describe globo.com’s live video stream architecture, which was used to broadcast events such as the FIFA World Cup (with peak of 500K concurrent users), Brazilian election debates (27 simultaneous streams) and BBB (10 cameras streaming 24/7 for 3 months) .

NGINX is one of the main components of our platform, as we use it for content distribution, caching, authentication, and dynamic content. Besides our architecture, we will also discuss the Nginx and Operational System tuning that was required for a 19Gbps throughput in each node, the open source Cassandra driver for Nginx that we developed, and our recent efforts to migrate to nginx-rtmp.

Great sources of info

Standing on the shoulders of giants

And for sure HN and SO.

FIFA 2014 World Cup live stream architecture

live_stream_nginx We were given the task to stream the FIFA 14 World Cup and I think this was an experience worth sharing. This is a quick overview about: the architecture, the components, the pain, the learning, the open source and etc.

The numbers

  • GER 7×1 BRA (yeah, we’re not proud of it)
  • 0.5M simultaneous users @ a single game – ARG x SUI
  • 580Gbps @ a single game – ARG x SUI
  • =~ 1600 watched years @ the whole event

The core overview

The project was to receive an input stream, generate HLS output stream for hundreds of thousands and to provide a great experience for final users:

  1. Fetch the RTMP input stream
  2. Generate HLS and send it to Cassandra
  3. Fetch binary and meta data from Cassandra and rebuild the HLS playlists with Nginx+lua
  4. Serve and cache the live content in a scalable way
  5. Design and implement the player

If you want to understand why we chose HLS check this presentation only in pt-BR. tip: sometimes we need to rebuild some things from scratch.

The input

The live stream comes to our servers as RTMP and we were using EvoStream (now we’re moving to nginx-rtmp) to receive this input and to generate HLS output to a known folder. Then we have some python daemons, running at the same machine, watching this known folder and parsing the m3u8 and posting the data to Cassandra.

To watch files modification and to be notified by these events, we first tried watchdog but for some reason we weren’t able to make it work as fast as we expected and we changed to pyinotify.

Another challenge we had to overcome was to make the python program scale to x cpu cores, we ended up by creating multiple Python processes and using async execution.

tip: maybe the best language / tool is in another castle.

The storage

We previously were using Redis to store the live stream data but we thought Cassandra was needed to offer DVR functionality easily (although we still uses Redis a lot). Cassandra response time was increasing with load to a certain point where clients started to timeout and the video playback completely stopped.

We were using it as Queue-like which turns out to be a anti-pattern. We then denormalized our data and also changed to LeveledCompactionStrategy as well as we set durable_writes to false, since we could treat our live stream as ephemeral data.

Finally, but most importantly, since we knew the maximum size a playlist could have, we could specify the start column (filtering with id > minTimeuuid(now – playlist_duration)). This really mitigated the effect of tombstones for reads. After these changes, we were able to achieve a latency in the order of 10ms for our 99% percentile.

tip: limit your queries + denormalize your data + send instrumentation data to graphite + use SSD.

The output

With all the data and meta-data we could build the HLS manifest and serve the video chunks. The only thing we were struggling was that we didn’t want to add an extra server to fetch and build the manifests.

Since we already had invested a lot of effort into Nginx+Lua, we thought it could be possible to use lua to fetch and build the manifest. It was a matter of building a lua driver for Cassandra and use it. One good thing about this approach (rebuilding the manifest) was that in the end we realized that we were almost ready to serve DASH.

tip: test your lua scripts + check the lua global vars + double check your caching config

The player

In order to provide a better experience, we chose to build Clappr, an extensible open-source HTML5 video player. With Clappr – and a few custom extensions like PiP (Picture In Picture) and Multi-angle replays – we were able to deliver a great experience to our users.

tip: open source it from day 0 + follow to flow issue -> commit FIX#123

The sauron

To keep an eye over all these system, we built a monitoring dashboard using mostly open source projects like: logstash, elastic search, graphite, graphana, kibana, seyren, angular, mongo, redis, rails and many others.

tip: use SSD for graphite and elasticsearch

The bonus round

Although we didn’t open sourced the entire solution, you can check most of them:


Discussion / QA @ HN

RSpec and Watir to test web applications

Testing is cool

Software testing

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation. Test techniques include, but are not limited to, the process of executing a program or application with the intent of finding software bugs (errors or other defects) – Wikipedia

The main intend of this post is, introduce you to UI tests over some ruby toys. In fact you could create an entire project (new) in ruby just to test your legacy web project. It’s cool, you can learn new language and work for the improvement of your legacy product. If you are totally new for ruby maybe a ruby overview can help you. (or might confuse you more)

Installing ruby, watir and rspec

Instead of installing the ruby directly, we are going to install the RVM (Ruby Version Manager) to then install any ruby we need. The steps described here were made on Ubuntu 11.04. On your terminal do the magic to install RVM.

bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
echo ‘[[ -s “$HOME/.rvm/scripts/rvm” ]] && . “$HOME/.rvm/scripts/rvm” # Load RVM function’ >> ~/.bash_profile
source .bash_profile

And from now on, your life will be better on ruby interpreters versions. Let’s install the ruby 1.9.2. (terminal again)

rvm install 1.9.2

And if we want to see the rubies installed on our machine?

rvm list

And now, how can we chose one ruby to work on the terminal session?

rvm use 1.9.2

For the test purpose we will use Watir and RSpec, great tools for testing, make fun with BDD and the best thing is install them it’s very easy.

gem install watir-webdriver
gem install rspec

Hands-on

Since we have all things installed, we can move for the example. The feature I want to test is the search system of  Amazon. Being more precise, I want to search for ‘Brazil’ and see if the ‘Brazil on the Rise’ is within the results as I want to be sure when I search for ‘semnocao‘ the Amazon doesn’t provide any result. Now, we can write the spec.

require 'amazon_page'

describe AmazonPage do
 before(:each) do
   @page = AmazonPage.new
 end
 after(:each) do
   @page.close
 end
 it "should show 'Brazil on the Rise' when I query for [Brazil]" do
  @page.query 'Brazil'
  @page.has_text('Brazil on the Rise').should == true
 end
 it "should bring no result when I search for [semnocao]" do
  @page.query 'semnocao'
  @page.results_count.should == 0
 end
end

The specification is very simple, it will create a page before each test calling and close the page after each test calling. There is only two tests: test when you search for Brazil  and  when you search for semnocao. We will design the tests using page object pattern. The class bellow is the page which represents the Amazon page and all testable behaviors should be inside of it.

require 'watir-webdriver'

class AmazonPage
 def initialize
  @page = Watir::Browser::new :firefox
  @page.goto 'http://www.amazon.com'
 end
 def close
  @page.quit
 end
 def query(parameter)
  @page.text_field(:id=>'twotabsearchtextbox').set parameter
  @page.send_keys :enter
 end
 def has_text(text)
  @page.text.include? text
 end
 def results_count
    if @page.text.include? 'did not match'
     0
    else
     @page.div(:id=>'resultCount').text.split(' ')[5].gsub(',','').to_i
    end
 end
end

To run this you just need to type on your terminal.

rspec spec/

The final code can be downloaded or viewed at github.

Additions

  • We could improve our story readibility  with Cucumber.
  • We could send the browser execution to an Xvfb server. (A.K.A. running headless) The browser pops up really bothers me.
  • We could integrate it with our CI.
  • We could design a base Page class for provide common operations as mixin or something

ps: the post was very inspired by KK post and Saush.

Acceptance testing on Fitnesse using slim test system

What is Fitnesse?

  • It’s a software development collaboration tool.
  • It’s a software testing tool.
  • It’s a wiki.
  • It’s a web server.

This tool provides a way for the BA’s and/or customers write their acceptance testing on wikis and better than this, they can run theirs tests and see if it fails or pass right on the page. Install Fitnesse it’s very simple, you just download and execute. The fitnesse architecture provides two types of test system: slim and the well-known fit. For this tutorial I’ll use the slim.

The hands-on

Scenario: Given that I have the input1 and input2 Then the output Should Be input1 plus space input2. So let’s express the acceptance testing of this feature (or behavior if you want). In Fitnesse we can express this using the decision table, others examples of places to write tests are query table, script table, library table and etc.

|it should print guaqmire|
|input1|input2|output?|
|you|are|you are|
|family|guy|family guy|
|tests|enough|tests enough|

Explaining the table, the first line is the name of the fixture, the second line contains the inputs and outputs names and from the third line on it’s filled with testing data. The output table should look like this:
Let’s setup the wiki-page to it became runnable, edit the page and put these parameters on the page.

!define TEST_SYSTEM {slim}
!define TEST_RUNNER {C:\fit\slim\rubyslim\lib\run_ruby_slim.rb}
!define PATH_SEPARATOR { -I }
!define COMMAND_PATTERN {ruby -I %p %m}
!path C:\fit\slim\rubyslim\lib\
!path C:\fit\ruby\prj\

The first line is setting TEST_SYSTEM to configure fitnesse to use the slim protocol instead of default fit. As we will use slim and ruby, we’ll use the rubyslim library. The second line is setting the TEST_RUNNER to use the ruby slim. Third line defines the PATH_SEPARATOR, used by the COMMAND_PATTERN to separate paths. The fourth line is configuring the COMMAND_PATTERN that will execute the test itself, the two parameters %p (receives all the paths from the page and its ancestors) and %m (the fixture itself) are used to correctly perform the test. The lines with path just informs to fitnesse where it can found the libraries and the runtime files.

And now you can run, is it fails? Good, now let’s programming it in ruby, if you are a newbie ruby as me, your code might be something like this.

module Fixtures
  class ItShouldPrintGuaqmire
   def set_input1 input1
    @input1 = input1
   end
   def set_input2 input2
    @input2 = input2
   end
   def output
    "#{@input1} #{@input2}"
   end
  end
end

Ohh it continues to fail, shame on me. As you can see at the code, I put the fixture inside a module called Fixture, so we need to inform the fitness what module/package is my fixture and we can do that by a table.

|Import|
|Fixtures|

This special table only  configures where is the fixtures. Now let’s see the entire code for fitnesse wiki.

!define TEST_SYSTEM {slim}
!define TEST_RUNNER {C:\fit\slim\rubyslim\lib\run_ruby_slim.rb}
!define PATH_SEPARATOR { -I }
!define COMMAND_PATTERN {ruby -I %p %m}
!path C:\fit\slim\rubyslim\lib\
!path C:\fit\ruby\prj\

|Import|
|Fixtures|

|it should print guaqmire|
|input1|input2|output?|
|you|are|you are|
|family|guy|family guy|
|tests|enough|tests enough|

Running the tests should show

Bonus round – Fitnesse using slim protocol in Java

In fact to make it runnable in Java it’s easier, you don’t need any TEST_RUNNER or COMMAND_PATTERN in your wiki page, since Java it’s default for fitnesse and your final wiki should look like this:

!define TEST_SYSTEM {slim}
!path C:\fit\java\prj\fit-slim-java.jar

|Import|
|br.com.leandromoreira.fixtures|

|it should print guaqmire|
|input1|input2|output?|
|you|are|you are|
|family|guy|family guy|
|tests|enough|tests enough|

And your Java code can be something like this:

package br.com.leandromoreira.fixtures;

public class ItShouldPrintGuaqmire{
  private String input1;
  private String input2;
  public void setInput1(final String i1){
   input1 = i1;
  }
  public void setInput2(final String i2){
   input2 = i2;
  }
  public String output(){
   return input1 + " " + input2;
  }
}

Real world

Usually the real world projects requires a lots of libraries on path, setup pages and more than just decision table to write tests. For instance, you can see that use fitnesse with slim seems more portable , less coupled with runtime and easier to implement too. In the real world you also create wiki for the project and suite test page for stories and organize all your imports and configs on project level, when you are composing a wiki on fitnesse you can take advantage of the fact that all the pages extend the configs from theirs ancestors, so you can have a better project wiki and managable test suite pages.

Update – Issues with Ruby 1.9.x

If you are trying to use the ruby 1.9.x you will have some issues the first one is: require ‘jcode’ issue, I tried to solve it but then it started to show another error list_deserializer.rb:1:in `<‘: comparison of String with Float failed (ArgumentError)’.  Since I’m not (still) a ruby guy I don’t know how to fix it.

Thoughtworks Brazil – POA

Thoughtworks logo
Now it’s official, I’m working on ThoughtWorks – a global IT consultancy which focuses on agile software development, has contributed to a range of open source products, including CruiseControl, NUnit 2.0, and Selenium. My last work was at Sefaz-GO, a government agency which mainly takes care of state tax system. I left too many friends there but this opportunity cames to me and I need to move on, I mean, literally move, from GYN to POA, that’s the worst part to me. Soon I’ll posting more about this new position in my career. Ohhh and like any new thing on life, I have my hopes: increase my knowledge, to know more cultures, apply agile with some of the creators and learn with amazing twkrs. And all this started around four months ago. I’m happy! 😀

For those are feeling lost please read thoughtworks at Brazil  and remember we’re continue hiring people.