Refactoring Ruby with Monads

Stacks

I’ll start with a rhetorical question: what’s a stack?

Well, it’s a kind of value that supports certain operations:

• #push(value)
• #pop
• #top
• #empty?
• .empty

I’m talking about an immutable stack here, so #push and #pop don’t mutate the stack, they just return a new one; you have to use #top to look at the top element. In addition to the four instance methods, we also need a class method called .empty which creates an empty stack to get us started.

For a value to qualify as a stack, its operations have to follow certain rules. For any stack,

• stack.push(x).top must equal x (regardless of what x is);
• stack.push(x).pop must equal stack;
• empty.empty? must be true; and
• stack.push(x).empty? must be false.

The rules just say that the operations do what you intuitively expect: when you push a value onto a stack, that becomes the top value; pushing then popping is a no-op; an empty stack is empty; and a stack with something pushed onto it isn’t empty.

We can implement the five stack operations however we like, as long as our implementation satisfies the rules.

Here’s a class called ArrayStack which implements the operations in a particular way:

ArrayStack = Struct.new(:values) do
  def push(value)
    ArrayStack.new([value] + values)
  end

  def top
    values.first
  end

  def pop
    ArrayStack.new(values.drop(1))
  end

  def empty?
    values.empty?
  end

  def self.empty
    new([])
  end
end

ArrayStack stores the stack contents as an array of values. ArrayStack#push adds a value onto the front of the values array, and ArrayStack#pop removes the array’s first item.

Here’s another class, LinkedListStack, which implements the same operations in a different way:

LinkedListStack = Struct.new(:top, :pop) do
  def push(value)
    LinkedListStack.new(value, self)
  end

  # def top
  #   # we get this for free!
  # end

  # def pop
  #   # and this!
  # end

  def empty?
    pop.nil?
  end

  def self.empty
    new(nil, nil)
  end
end

Instead of storing a complete array of values, LinkedListStack stores the top element and a pointer to the rest of the stack. LinkedListStack#top and LinkedListStack#pop are simple getters, and LinkedListStack#push stores its value in a new LinkedListStack instance pointing at the old one.

So, what’s the point of the rules? Well, they guarantee how the stack operations will behave, so once those operations have been implemented in a compliant way, we can use them without knowing anything about their implementation.

For example, if we make an empty ArrayStack and push two values onto it, then pop one of them and ask for the top value, we get the result we expect:

>> stack = ArrayStack.empty
=> #<struct ArrayStack values=[]>

>> stack.push('hello').push('world').pop.top
=> "hello"

If we do the same with a LinkedListStack, it works in exactly the same way, because the same rules apply:

>> stack = LinkedListStack.empty
=> #<struct LinkedListStack top=nil, rest=nil>

>> stack.push('hello').push('world').pop.top
=> "hello"

The guarantee provided by the rules also makes it possible for us to define new operations in terms of the old ones.

For example, here’s a method called #size that recursively calculates the size of a stack by counting how many times it has to be popped until it’s empty:

module Stack
  def size
    if empty?
      0
    else
      pop.size + 1
    end
  end
end

ArrayStack.include(Stack)
LinkedListStack.include(Stack)

As long as a stack has compliant implementations of #empty? and #pop, we can implement #size on top of them. It doesn’t matter what stack implementation we use; #size always works the same.

An empty ArrayStack has size 0, and if we push two values on, the resulting stack’s size is 2:

>> ArrayStack.empty.size
=> 0

>> ArrayStack.empty.push('hello').push('world').size
=> 2

Likewise, an empty LinkedListStack has size 0, and if we push two values on, the resulting stack’s size is 2:

>> LinkedListStack.empty.size
=> 0

>> LinkedListStack.empty.push('hello').push('world').size
=> 2

So what do we mean by “stack”? It’s really a specification. An implementation of a stack provides certain operations that follow certain rules.

The stack specification has two benefits: firstly, those operations provide a common interface across many possible implementations; and secondly, they allow us to build shared functionality (like the #size method) that works with any compliant implementation.

Collections

Here’s another question: what’s a collection (in Ruby, at least)?

Again, it’s a kind of value with certain operations — actually just one operation, called #each. That operation follows certain rules — actually just one rule, which is that #each must call a block with a value zero or more times in immediate sequence. That’s it.

We can implement that #each operation however we like, as long as we obey the rule. Here’s a HardcodedCollection class whose #each method literally calls a block with 1, then with 2, all the way up to 5:

class HardcodedCollection
  def each(&block)
    block.call(1)
    block.call(2)
    block.call(3)
    block.call(4)
    block.call(5)
  end
end

Here’s a GeneratedCollection whose #each method calculates those values dynamically in a loop and calls a block with each one:

class GeneratedCollection
  def each(&block)
    number = 1
    while number <= 5
      block.call(number)
      number = number + 1
    end
  end
end

We can use #each without knowing its implementation. From the outside, HardcodedCollection and GeneratedCollection both behave like a collection of the numbers from 1 to 5:

>> collection = HardcodedCollection.new
=> #<HardcodedCollection>

>> collection.each { |n| puts n }
1
2
3
4
5
=> nil

>> collection = GeneratedCollection.new
=> #<GeneratedCollection>

>> collection.each { |n| puts n }
1
2
3
4
5
=> nil

Just as we did with stacks, we can define more collection operations on top of the one operation we already have. Here’s a method called #select that takes a block, then calls #each and accumulates all the values that make the block return true:

module Collection
  def select(&block)
    results = []
    each do |value|
      results << value if block.call(value)
    end
    results
  end
end

HardcodedCollection.include(Collection)
GeneratedCollection.include(Collection)

As long as a collection has a working implementation of #each, we can implement #select on top of it. It works the same for both implementations:

>> HardcodedCollection.new.select { |n| n.odd? }
=> [1, 3, 5]

>> GeneratedCollection.new.select { |n| n.odd? }
=> [1, 3, 5]

Of course, in Ruby we have a module called Enumerable that already implements #select, as well as #count, #map, #inject, and tons of other helpful stuff that all sits on top of this one #each method.

So, what do we mean by “collection”? Again, it’s really just a specification. An implementation of a collection provides an operation that follows a rule.

And again, we get two benefits: the #each method gives us a common interface across many possible implementations of collections, and it allows us to build shared functionality (like #select) that works with any implementation.

Abstract data types

We’ve looked at stacks and collections, and seen some similarities between them. What name do we give this kind of thing? Are “stack” and “collection” design patterns? Interfaces? APIs? Duck types?

Those words are all appropriate, and they overlap to an extent; the concept of a “stack” or a “collection” sits in the middle of all of them. In my opinion, the most specific (and therefore most accurate) term is “abstract data type”, which literally means a kind of value with certain operations that follow certain rules.

So stacks and collections are abstract concepts, but they’re abstract in a good way! The abstractions give us power, and we expect programmers to understand them. Nobody talks about stacks in hushed tones. They’re simple and useful.

Handling nil

First, I’d like to look at some code that has to deal with nils.

Imagine that we have a project management application with different kinds of models:

Project = Struct.new(:creator)
Person  = Struct.new(:address)
Address = Struct.new(:country)
Country = Struct.new(:capital)
City    = Struct.new(:weather)

Each Project has a Person who created it; each Person has an Address; each Address has a Country; each Country has a capital City; and each City has weather information, which, for the sake of simplicity, is just a string.

Let’s say that we want to display the weather next to each project in our user interface (for some reason). That involves traversing all these associations. Here’s a method that does that:

def weather_for(project)
  project.creator.address.
    country.capital.weather
end

(Maybe you’ve written similar Rails view helpers before. There are lots of reasons not to write code like this, but there are also perfectly good reasons to do it, and anyway, people will always write code like this no matter what we say.)

If we make a city which has sunny weather, and a country which has that city as its capital, and an address in that country, and a person with that address, and a project created by that person…

>> city = City.new('sunny')
=> #<struct City …>

>> country = Country.new(city)
=> #<struct Country …>

>> address = Address.new(country)
=> #<struct Address …>

>> person = Person.new(address)
=> #<struct Person …>

>> project = Project.new(person)
=> #<struct Project …>

…then we can pass that project into #weather_for and it works fine:

>> weather_for(project)
=> "sunny"

But if we make a bad project, for example by providing an address that has no country, #weather_for blows up:

>> bad_project = Project.new(Person.new(Address.new(nil)))
=> #<struct Project …>

>> weather_for(bad_project)
NoMethodError: undefined method `capital' for nil:NilClass

Tony Hoare invented nil in 1965; he now calls it his “billion-dollar mistake”, which has “probably caused a billion dollars of pain and damage”. This is exactly the sort of thing he’s talking about.

Well, they may be a mistake, but Ruby has nils, so we’re stuck with them. To make #weather_for tolerate nils, we’re going to have to explicitly check for them.

First we need to introduce local variables to hold every intermediate result…

def weather_for(project)
  creator = project.creator
  address = creator.address
  country = address.country
  capital = country.capital
  weather = capital.weather
end

…and then check each intermediate result before we try to call a method on it:

def weather_for(project)
  unless project.nil?
    creator = project.creator
    unless creator.nil?
      address = creator.address
      unless address.nil?
        country = address.country
        unless country.nil?
          capital = country.capital
          unless capital.nil?
            weather = capital.weather
          end
        end
      end
    end
  end
end

(While we’re at it, we might as well include the possibility that the project itself is nil.)

The method body is starting to drift right and become a pyramid of doom, but luckily it works the same if we flatten it:

def weather_for(project)
  unless project.nil?
    creator = project.creator
  end

  unless creator.nil?
    address = creator.address
  end

  unless address.nil?
    country = address.country
  end

  unless country.nil?
    capital = country.capital
  end

  unless capital.nil?
    weather = capital.weather
  end
end

This code works, but it’s pretty clumsy, and it’s hard to remember to do something like this every time we might possibly have nils to deal with.

Fortunately, Ruby on Rails has a solution to this problem. Rails (actually Active Support) monkey patches Object and NilClass with a method called #try, which delegates to #public_send if the object’s not nil, and returns nil otherwise:

class Object
  def try(*a, &b)
    if a.empty? && block_given?
      yield self
    else
      public_send(*a, &b) if respond_to?(a.first)
    end
  end
end

class NilClass
  def try(*args)
    nil
  end
end

When every object has a #try method, instead of doing these nil checks ourselves, we can let #try do it for us:

def weather_for(project)
  creator = project.try(:creator)
  address = creator.try(:address)
  country = address.try(:country)
  capital = country.try(:capital)
  weather = capital.try(:weather)
end

Now we’re back to just chaining method calls together, so we can take the local variables out again:

def weather_for(project)
  project.
    try(:creator).
    try(:address).
    try(:country).
    try(:capital).
    try(:weather)
end

This is good as it gets right now — better than the version with “unless nil?” all over the place, at least. Can we do any better?

Well, monkey patching has its place, but monkey patching every single object in the system isn’t great. It’s kind of a code smell, so let’s not do it.

When we want to add a method to an object, the good object-oriented programming solution is to use decoration, where we non-invasively add functionality to one object by wrapping it up inside another object.

Let’s make a decorator class, Optional, whose instances have a single attribute called value:

Optional = Struct.new(:value)

Instances of this class just wrap up another value. We can wrap a value like 'hello', then take 'hello' out again later:

>> optional_string = Optional.new('hello')
=> #<struct Optional value="hello">

>> optional_string.value
=> "hello"

If the value we put in happens to be nil, we get nil out later:

>> optional_string = Optional.new(nil)
=> #<struct Optional value=nil>

>> optional_string.value
=> nil

So now, instead of putting the #try method on Object, let’s put it on Optional:

class Optional
  def try(*args, &block)
    if value.nil?
      nil
    else
      value.public_send(*args, &block)
    end
  end
end

If the value attribute is nil, #try just returns nil, otherwise it sends the appropriate message to the underlying object.

Now we can call #try on the decorator and it’ll call the method on the underlying object as long as it’s not nil:

>> optional_string = Optional.new('hello')
=> #<struct Optional value="hello">

>> length = optional_string.try(:length)
=> 5

If the value inside the Optional is nil, #try will just return nil:

>> optional_string = Optional.new(nil)
=> #<struct Optional value=nil>

>> length = optional_string.try(:length)
=> nil

So instead of calling #try on the actual project object, and then on the actual person object, and so on…

def weather_for(project)
  creator = project.try(:creator)
  address = creator.try(:address)
  country = address.try(:country)
  capital = country.try(:capital)
  weather = capital.try(:weather)
end

…we can write the method like this:

def weather_for(project)
  optional_project = Optional.new(project)
  optional_creator = Optional.new(optional_project.try(:creator))
  optional_address = Optional.new(optional_creator.try(:address))
  optional_country = Optional.new(optional_address.try(:country))
  optional_capital = Optional.new(optional_country.try(:capital))
  optional_weather = Optional.new(optional_capital.try(:weather))
  weather          = optional_weather.value
end

First we decorate project with an Optional object, and call #try on that. Then we decorate the result, which might be nil, and call #try again. Then we decorate the next result, and call #try on that, and so on. At the end, we pull out the value and return it.

It’s unwieldy, but hey, at least we’re not monkey patching every object in the system.

There’s another code smell here: #try does too much. We actually just wanted to refactor away the nil check, but #try also sends the value a message. What if we want to use the value in some other way when it’s not nil? Our #try method is overspecialised; it has too much responsibility.

Instead of hard-coding the else clause inside #try, let’s allow its caller to supply a block that controls what happens next:

class Optional
  def try(&block)
    if value.nil?
      nil
    else
      block.call(value)
    end
  end
end

Now we can pass a block to #try, and do whatever we want with the underlying value: send it a message, or use it as an argument in a method call, or print it out, or whatever. (This ability to supply a block is actually a little-used feature of Active Support’s #try too.)

So now, instead of calling #try with a message name, and having to remember that it’s going to send that message to the underlying object, we call it with a block, and in the block we send the message ourselves and decorate the result in an Optional:

def weather_for(project)
  optional_project = Optional.new(project)
  optional_creator = optional_project.try { |project| Optional.new(project.creator) }
  optional_address = optional_creator.try { |creator| Optional.new(creator.address) }
  optional_country = optional_address.try { |address| Optional.new(address.country) }
  optional_capital = optional_country.try { |country| Optional.new(country.capital) }
  optional_weather = optional_capital.try { |capital| Optional.new(capital.weather) }
  weather          = optional_weather.value
end

And we’re now able to do whatever we want with the value, like print it out in a log message.

That works fine when there aren’t any nils, but unfortunately we’ve broken it when nils are involved, because we’re returning nil when the block doesn’t run:

>> weather_for(project)
=> "sunny"

>> weather_for(bad_project)
NoMethodError: undefined method `capital' for nil:NilClass

That’s easy to fix. Instead of returning a raw nil, we’ll decorate it with an Optional first:

class Optional
  def try(&block)
    if value.nil?
      Optional.new(nil)
    else
      block.call(value)
    end
  end
end

And now it works in both cases:

>> weather_for(project)
=> "sunny"

>> weather_for(bad_project)
=> nil

But there’s a new code smell: I don’t think #try is a great name any more, because we’ve changed it to do something more general than, or at least something different from, the main use case of its Active Support namesake.

Let‘s rename it to #and_then:

class Optional
  def and_then(&block)
    if value.nil?
      Optional.new(nil)
    else
      block.call(value)
    end
  end
end

Because it really just says, “start with this decorated value, and then do some arbitrary thing with it, as long as it’s not nil”.

Here’s the new version of #weather_for, which calls #and_then instead of #try:

def weather_for(project)
  optional_project = Optional.new(project)
  optional_creator = optional_project.and_then { |project| Optional.new(project.creator) }
  optional_address = optional_creator.and_then { |creator| Optional.new(creator.address) }
  optional_country = optional_address.and_then { |address| Optional.new(address.country) }
  optional_capital = optional_country.and_then { |country| Optional.new(country.capital) }
  optional_weather = optional_capital.and_then { |capital| Optional.new(capital.weather) }
  weather          = optional_weather.value
end

And because we’re just chaining #and_then calls, we can get rid of the local variables again:

def weather_for(project)
  Optional.new(project).
    and_then { |project| Optional.new(project.creator) }.
    and_then { |creator| Optional.new(creator.address) }.
    and_then { |address| Optional.new(address.country) }.
    and_then { |country| Optional.new(country.capital) }.
    and_then { |capital| Optional.new(capital.weather) }.
    value
end

This is verbose but nice: we decorate the (possibly nil) project in an Optional object, then safely traverse all the associations, then pull the (possibly nil) value out again at the end.

Phew, okay. How’s our refactoring going?

Well, we might not be monkey patching anything, and it’s conceptually clean, but there’s a huge final smell: nobody wants to write code like this! In theory it might be better than Active Support’s #try method, but in practice it’s worse.

But we can add some syntactic sugar to fix that. Here’s a definition of #method_missing for Optional:

class Optional
  def method_missing(*args, &block)
    and_then do |value|
      Optional.new(value.public_send(*args, &block))
    end
  end
end

It uses #and_then to delegate any message to the underlying value whenever it’s not nil. Now we can replace all of the “and_then … Optional.new” stuff with just normal message sends, and let #method_missing take care of the details:

def weather_for(project)
  Optional.new(project).
    creator.address.country.capital.weather.
    value
end

This is actually really good! You can see very clearly that we wrap up the possibly-nil project into an Optional, then safely perform our chain of method calls, then pull the possibly-nil weather out of an Optional at the end.

To recap, here’s the full definition of Optional:

Optional = Struct.new(:value) do
  def and_then(&block)
    if value.nil?
      Optional.new(nil)
    else
      block.call(value)
    end
  end

  def method_missing(*args, &block)
    and_then do |value|
      Optional.new(value.public_send(*args, &block))
    end
  end
end

We designed an object which stores a value that might be nil, and a method called #and_then which encapsulates the nil-check logic. We added some sugar on top by writing #method_missing. (If this was production code, we should remember to implement #respond_to? as well.)

I’d like to very briefly point out that we only need to do the decorating and undecorating for compatibility with the rest of the system. If the rest of the system passed in an Optional and expected us to return one, we wouldn’t even need to do that:

def weather_for(project)
  project.creator.address.
    country.capital.weather
end

And then we wouldn’t have to remember to check for nil at all! We could write the method the way we did in the first place and it would just work. Imagine that.

Multiple results

Alright, that refactoring was very detailed. We’re going to do two more, but we’ll skip the detail to keep things manageable. Let’s refactor some code that has to handle multiple results.

Imagine we have a content management application with different kinds of models:

Blog     = Struct.new(:categories)
Category = Struct.new(:posts)
Post     = Struct.new(:comments)

There are several Blogs; each Blog has many Categorys; each Category has many Posts; and each Post has many comments, which, for the sake of simplicity, are just strings.

Let’s say that we want to fetch all the words from all the comments within certain blogs (for some reason). That involves traversing all these associations.

Here’s a method that does that:

def words_in(blogs)
  blogs.flat_map { |blog|
    blog.categories.flat_map { |category|
      category.posts.flat_map { |post|
        post.comments.flat_map { |comment|
          comment.split(/\s+/)
        }
      }
    }
  }
end

At each level we map over a collection and traverse the association for each object inside it. When we reach each comment, we split it on whitespace to get its words. We have to use #flat_map because we want a flattened array of words instead of a nested one.

If we make a couple of blogs, which each have a couple of categories, which contain some posts, which have some comments, which contain some words…

blogs = [
  Blog.new([
    Category.new([
      Post.new(['I love cats', 'I love dogs']),
      Post.new(['I love mice', 'I love pigs'])
    ]),
    Category.new([
      Post.new(['I hate cats', 'I hate dogs']),
      Post.new(['I hate mice', 'I hate pigs'])
    ])
  ]),
  Blog.new([
    Category.new([
      Post.new(['Red is better than blue'])
    ]),
    Category.new([
      Post.new(['Blue is better than red'])
    ])
  ])
]

…then #words_in can extract all of the words:

>> words_in(blogs)
=> ["I", "love", "cats", "I", "love", "dogs", "I",
    "love", "mice", "I", "love", "pigs", "I",
    "hate", "cats", "I", "hate", "dogs", "I",
    "hate", "mice", "I", "hate", "pigs", "Red",
    "is", "better", "than", "blue", "Blue", "is",
    "better", "than", "red"]

But #words_in has a bit of a pyramid of doom going on, plus it’s hard to distinguish between the code doing actual work and the boilerplate of dealing with multiple values.

We can clean it up by introducing a class, Many, whose instances decorate a collection of values:

Many = Struct.new(:values) do
  def and_then(&block)
    Many.new(values.map(&block).flat_map(&:values))
  end
end

Like Optional, Many has an #and_then method that takes a block, but this time it calls the block for every value in the collection and flattens the results together.

Now we can replace all of #words_in’s calls to #flat_map with instances of Many and calls to #and_then:

def words_in(blogs)
  Many.new(blogs).and_then do |blog|
    Many.new(blog.categories).and_then do |category|
      Many.new(category.posts).and_then do |post|
        Many.new(post.comments).and_then do |comment|
          Many.new(comment.split(/\s+/))
        end
      end
    end
  end.values
end

Now we can flatten the pyramid…

def words_in(blogs)
  Many.new(blogs).and_then do |blog|
    Many.new(blog.categories)
  end.and_then do |category|
    Many.new(category.posts)
  end.and_then do |post|
    Many.new(post.comments)
  end.and_then do |comment|
    Many.new(comment.split(/\s+/))
  end.values
end

…and reformat the code a little to get this:

def words_in(blogs)
  Many.new(blogs).
    and_then { |blog    | Many.new(blog.categories)      }.
    and_then { |category| Many.new(category.posts)       }.
    and_then { |post    | Many.new(post.comments)        }.
    and_then { |comment | Many.new(comment.split(/\s+/)) }.
    values
end

Again, this is pretty clear, but we can add some syntactic sugar by defining #method_missing:

class Many
  def method_missing(*args, &block)
    and_then do |value|
      Many.new(value.public_send(*args, &block))
    end
  end
end

This is exactly the same as the Optional#method_missing, except it calls Many.new instead of Optional.new.

Now we can replace all of the “and_then … Many.new” calls with simple message sends:

def words_in(blogs)
  Many.new(blogs).
    categories.posts.comments.split(/\s+/).
    values
end

This is very nice! We put the blog posts into a Many object, traverse all the associations, then take the values out at the end.

And again, if the rest of the system could deal with instances of Many, we could just expect one and return one:

def words_in(blogs)
  blogs.categories.posts.comments.split(/\s+/)
end

To recap, here’s the class we just made:

Many = Struct.new(:values) do
  def and_then(&block)
    Many.new(values.map(&block).flat_map(&:values))
  end

  def method_missing(*args, &block)
    and_then do |value|
      Many.new(value.public_send(*args, &block))
    end
  end
end

Asynchronous code

For our third quick refactoring, we’re going to tackle writing asynchronous code.

I’ve often wondered who the most influential Rubyist is. Let’s find out once and for all, by using the GitHub API to find the person who’s made the most commits on the most popular Ruby project.

When you make an HTTP GET request to the GitHub API root, you get back some JSON that looks more or less like this:

GET https://api.github.com/

{
  "current_user_url":      "https://api.github.com/user",
  "authorizations_url":    "https://api.github.com/authorizations",
  "emails_url":            "https://api.github.com/user/emails",
  "emojis_url":            "https://api.github.com/emojis",
  "events_url":            "https://api.github.com/events",
  "feeds_url":             "https://api.github.com/feeds",
  "following_url":         "https://api.github.com/user/following{/target}",
  "gists_url":             "https://api.github.com/gists{/gist_id}",
  "hub_url":               "https://api.github.com/hub",
  "issues_url":            "https://api.github.com/issues",
  "keys_url":              "https://api.github.com/user/keys",
  "notifications_url":     "https://api.github.com/notifications",
  "organization_url":      "https://api.github.com/orgs/{org}",
  "public_gists_url":      "https://api.github.com/gists/public",
  "rate_limit_url":        "https://api.github.com/rate_limit",
  "repository_url":        "https://api.github.com/repos/{owner}/{repo}",
  "starred_url":           "https://api.github.com/user/starred{/owner}{/repo}",
  "starred_gists_url":     "https://api.github.com/gists/starred",
  "team_url":              "https://api.github.com/teams",
  "user_url":              "https://api.github.com/users/{user}"
}

Among other things, this gives us a URI template for finding out information about any organisation. Now we know what URL to use to get info about the Ruby organisation:

GET https://api.github.com/orgs/ruby

When we make a request to this URL, we get some JSON that contains a URL we can use to get a list of all the Ruby organisation’s repositories. So we fetch the list of repositories, which includes information about how many watchers each one has:

GET https://api.github.com/orgs/ruby/repos

From that we can see which repo has the most watchers (the main Ruby repo) and the URL for that repo’s representation in the API:

GET https://api.github.com/repos/ruby/ruby

When we fetch that repository’s information, we get another URL that tells us where to get its list of contributors:

GET https://api.github.com/repos/ruby/ruby/contributors

So then we can load the list of contributors to the main Ruby repo, which includes information about how many commits each contributor has made. We pick the one with the most commits, a user called “nobu”, and finally fetch information about “nobu” from the URL in the contributor list:

GET https://api.github.com/users/nobu

{
  "login": "nobu",
  …
  "name": "Nobuyoshi Nakada",
  …
}

It turns out that Nobuyoshi Nakada has made the most commits on the most popular Ruby project. Thanks Nobuyoshi!

Okay, that was exhausting, so let’s write some code to do it for us.

Assume we already have this #get_json method:

def get_json(url, &success)
  Thread.new do
    uri   = URI.parse(url)
    json  = Net::HTTP.get(uri)
    value = JSON.parse(json)
    success.call(value)
  end
end

#get_json asynchronously makes an HTTP GET request, parses the JSON response into a Ruby hash or array, then calls a callback with the data. (Alternatively, you can imagine the single-threaded non-blocking EventMachine equivalent if you like.)

To do what we just did, we have to:

• get the URI templates from the GitHub API root;
• fill in the template with the name of the Ruby organisation;
• get the organisation data;
• find the URL for the list of its repositories;
• get the list of its repositories;
• find the URL of the repository with the most watchers;
• get the information on that repository;
• find the URL for the list of its contributors;
• get the list of its contributors;
• find the URL of the contributor with the most commits;
• get the information on that user; and finally
• print out their real name and username.

Here’s the code:

require 'uri_template'

get_json('https://api.github.com/') do |urls|
  org_url_template = URITemplate.new(urls['organization_url'])
  org_url = org_url_template.expand(org: 'ruby')

  get_json(org_url) do |org|
    repos_url = org['repos_url']

    get_json(repos_url) do |repos|
      most_popular_repo = repos.max_by { |repo| repo['watchers_count'] }
      repo_url = most_popular_repo['url']

      get_json(repo_url) do |repo|
        contributors_url = repo['contributors_url']

        get_json(contributors_url) do |users|
          most_prolific_user = users.max_by { |user| user['contributions'] }
          user_url = most_prolific_user['url']

          get_json(user_url) do |user|
            puts "The most influential Rubyist is #{user['name']} (#{user['login']})"
          end
        end
      end
    end
  end
end

This works, but it’s drifting right again. It’s hard to understand and maintain deeply nested code like this, but we can’t flatten it because of the nested callbacks.

Very briefly, the solution is to make an Eventually class that decorates a block:

Eventually = Struct.new(:block) do
  def initialize(&block)
    super(block)
  end

  def run(&success)
    block.call(success)
  end
end

The idea is that the block computes a value that might take a while to produce. Eventually#run runs the block with a callback for it to call when the value becomes available.

The gory details aren’t important, but here’s an #and_then method that we can use to add extra asynchronous processing to the value produced by an Eventually:

class Eventually
  def and_then(&block)
    Eventually.new do |success|
      run do |value|
        block.call(value).run(&success)
      end
    end
  end
end

This is more complicated than the other implementations of #and_then we’ve seen, but it achieves the same thing. (The difficult part is getting the callbacks wired up correctly.)

Now we can rewrite our code by putting each asynchronous #get_json call inside a block that we decorate with an Eventually object:

Eventually.new { |s| get_json('https://api.github.com/', &s) }.and_then do |urls|
  org_url_template = URITemplate.new(urls['organization_url'])
  org_url = org_url_template.expand(org: 'ruby')

  Eventually.new { |s| get_json(org_url, &s) }.and_then do |org|
    repos_url = org['repos_url']

    Eventually.new { |s| get_json(repos_url, &s) }.and_then do |repos|
      most_popular_repo = repos.max_by { |repo| repo['watchers_count'] }
      repo_url = most_popular_repo['url']

      Eventually.new { |s| get_json(repo_url, &s) }.and_then do |repo|
        contributors_url = repo['contributors_url']

        Eventually.new { |s| get_json(contributors_url, &s) }.and_then do |users|
          most_prolific_user = users.max_by { |user| user['contributions'] }
          user_url = most_prolific_user['url']

          Eventually.new { |s| get_json(user_url, &s) }
        end
      end
    end
  end
end.run do |user|
  puts "The most influential Rubyist is #{user['name']} (#{user['login']})"
end

We connect all the Eventuallys with #and_then, then #run them. This isn’t super readable either, but now we can pull out each logical part into its own method. The code that gets all the URL templates from GitHub can go into a method called #get_github_api_urls:

def get_github_api_urls
  github_root_url = 'https://api.github.com/'

  Eventually.new { |success| get_json(github_root_url, &success) }
end

This returns an Eventually which decorates a block that’ll eventually call its callback with the result of fetching and parsing the JSON.

So we can replace the line at the top of our code with “get_github_api_urls.and_then”:

get_github_api_urls.and_then do |urls|
  org_url_template = URITemplate.new(urls['organization_url'])
  org_url = org_url_template.expand(org: 'ruby')

  Eventually.new { |s| get_json(org_url, &s) }.and_then do |org|
    repos_url = org['repos_url']

    Eventually.new { |s| get_json(repos_url, &s) }.and_then do |repos|
      most_popular_repo = repos.max_by { |repo| repo['watchers_count'] }
      repo_url = most_popular_repo['url']

      Eventually.new { |s| get_json(repo_url, &s) }.and_then do |repo|
        contributors_url = repo['contributors_url']

        Eventually.new { |s| get_json(contributors_url, &s) }.and_then do |users|
          most_prolific_user = users.max_by { |user| user['contributions'] }
          user_url = most_prolific_user['url']

          Eventually.new { |s| get_json(user_url, &s) }
        end
      end
    end
  end
end.run do |user|
  puts "The most influential Rubyist is #{user['name']} (#{user['login']})"
end

This next bit of code that fetches the data for the Ruby organisation can go into a method called #get_org:

def get_org(urls, name)
  org_url_template = URITemplate.new(urls['organization_url'])
  org_url = org_url_template.expand(org: name)

  Eventually.new { |success| get_json(org_url, &success) }
end

This returns an Eventually object too.

So we can replace the next bit of code with a call to #get_org:

get_github_api_urls.and_then do |urls|
  get_org(urls, 'ruby').and_then do |org|
    repos_url = org['repos_url']

    Eventually.new { |s| get_json(repos_url, &s) }.and_then do |repos|
      most_popular_repo = repos.max_by { |repo| repo['watchers_count'] }
      repo_url = most_popular_repo['url']

      Eventually.new { |s| get_json(repo_url, &s) }.and_then do |repo|
        contributors_url = repo['contributors_url']

        Eventually.new { |s| get_json(contributors_url, &s) }.and_then do |users|
          most_prolific_user = users.max_by { |user| user['contributions'] }
          user_url = most_prolific_user['url']

          Eventually.new { |s| get_json(user_url, &s) }
        end
      end
    end
  end
end.run do |user|
  puts "The most influential Rubyist is #{user['name']} (#{user['login']})"
end

The code that gets all of the Ruby organisation’s repositories can go into a #get_repos method:

def get_repos(org)
  repos_url = org['repos_url']

  Eventually.new { |success| get_json(repos_url, &success) }
end

And then we can call it:

get_github_api_urls.and_then do |urls|
  get_org(urls, 'ruby').and_then do |org|
    get_repos(org).and_then do |repos|
      most_popular_repo = repos.max_by { |repo| repo['watchers_count'] }
      repo_url = most_popular_repo['url']

      Eventually.new { |s| get_json(repo_url, &s) }.and_then do |repo|
        contributors_url = repo['contributors_url']

        Eventually.new { |s| get_json(contributors_url, &s) }.and_then do |users|
          most_prolific_user = users.max_by { |user| user['contributions'] }
          user_url = most_prolific_user['url']

          Eventually.new { |s| get_json(user_url, &s) }
        end
      end
    end
  end
end.run do |user|
  puts "The most influential Rubyist is #{user['name']} (#{user['login']})"
end

And so on for the rest of it:

get_github_api_urls.and_then do |urls|
  get_org(urls, 'ruby').and_then do |org|
    get_repos(org).and_then do |repos|
      get_most_popular_repo(repos).and_then do |repo|
        get_contributors(repo).and_then do |users|
          get_most_prolific_user(users)
        end
      end
    end
  end
end.run do |user|
  puts "The most influential Rubyist is #{user['name']} (#{user['login']})"
end

Now that we’re just creating Eventually objects at each step, we don’t need to call #and_then on each one immediately. We can let each object be returned from its enclosing block before we call #and_then on it.

In other words, we can flatten the nested blocks to get this:

get_github_api_urls.and_then do |urls|
  get_org(urls, 'ruby')
end.and_then do |org|
  get_repos(org)
end.and_then do |repos|
  get_most_popular_repo(repos)
end.and_then do |repo|
  get_contributors(repo)
end.and_then do |users|
  get_most_prolific_user(users)
end.run do |user|
  puts "The most influential Rubyist is #{user['name']} (#{user['login']})"
end

I’ll just reformat that:

get_github_api_urls.
  and_then { |urls | get_org(urls, 'ruby')         }.
  and_then { |org  | get_repos(org)                }.
  and_then { |repos| get_most_popular_repo(repos)  }.
  and_then { |repo | get_contributors(repo)        }.
  and_then { |users| get_most_prolific_user(users) }.
  run do |user|
    puts "The most influential Rubyist is #{user['name']} (#{user['login']})"
  end

This is much nicer than what we had before. Each part is nicely encapsulated in its own method, and the parts are connected together in a clean way. This might be a familiar pattern: it’s similar to deferrables, promises and futures, which you may have seen in EventMachine, JavaScript or Clojure.

To recap, here’s the whole Eventually class:

Eventually = Struct.new(:block) do
  def initialize(&block)
    super(block)
  end

  def run(&success)
    block.call(success)
  end

  def and_then(&block)
    Eventually.new do |success|
      run do |value|
        block.call(value).run(&success)
      end
    end
  end
end