Principles That Will Make You Become a Badass Developer

In my last post, I went over things we should avoid if we want to become badass developers. As I said though, this is far from enough. Once we’ve stoped losing trust from the business, it’s time to build some more ! This is the forth post in a series about how to get sponsorship for a large scale refactoring. If you haven’t, start reading from the beginning.

The badass developers gain the business’s trust by sneaking in as business partners. A good way to become one is to start acting like one ! Here are examples of principles for that.

Arm of a badass developer with the tatoo '> Badass Principles;'

Keeping the business best interests in mind

This is all about decision making. We should try to steer decisions according to the business. Whenever we talk with business people, we should stay aways from technical topics. I still remember my younger self explaining threading details to a trader … Most of all I remember the look on his face ! We should avoid technical bla bla, but we should be clear about the business consequences.

Honesty and Candor

When we don’t agree with something, we should say so. When we don’t understand something, we should ask the question. We need to stick to facts and assume everyone has the business’s best interests in mind. Candor is a way to get our opinions and questions through, without sounding rude or pushy. There’s a whole chapter about candor in Creativity.inc, the book about Pixar.

Cover of the Creativity.inc book. It contains lessons on Candor we should all read to become badass developers

With time, business people will think of us as a positive and pragmatic problem solvers. That is exactly the kind of people they want to work with !

💡 Candor is a way to get our opinions and questions through, without sounding rude or pushy.

Strong opinions, but weekly held

Jeff Atwood, already wrote about this. The idea is to fight for our opinions, but let them go without a fuss when we proved wrong. We know that we are all very often wrong. Only fools or self-centered people don’t admit this reality. Business people won’t trust us in either case. We need to show that we can go over our previous opinions. This grows our reputation of rational problem solver.

Acknowledging when we don’t know

The same logic goes with knowledge. None of us knows everything. We have to admit when we don’t know something and ask for help. This proves that we place the business’s speed over our personal ‘know-it-all’ reputation.

Here is a story that happened to me at my first job. I’m sure most developers go through it one day or another. I was assigned a new task involving technologies I did not know. I did not have the guts to state upfront that I would have to learn them. The result was that I sent 2 full weeks fiddling with this task to get something working. The more it went on, the more the product people were wondering why it was taking so long, and the more I got stressed !

Be bold and say No !

If we are sure something we should not do something, we need to say so. Badass developers are not afraid to say they won’t do it. Good software engineering requires merciless prioritization. If there are doubt about the value of doing something, it’s often a better idea to make sure before wasting time on it.

There are many ways to say ‘No’. Before giving a harsh ‘No’, we can try to challenge decisions. We can ask for clarifications and rationals through open questions. Very often, highlighting the risks makes people review their plans. As technical experts, we should also share as much of the consequences as possible.

In the end, badass developers are ready to leave a FUBARed situation. Great engineers don’t have troubles finding jobs these days … There’s no reason they should accept to be waisting their time.

💡 In the end, badass developers are ready to leave a FUBARed situation

What do to next ?

As we become badass developers, our reputation will grow. We’ll be in a better position to negotiate a large scale refactoring with the business. There’s a catch though : we’ll need to live up to our reputation ! Admitting that we are wrong 100% with candor will not make it ! 

When we manage to negotiate a large scale refactoring, the team will need to do a good job of it. This boils down to delivering it piece by piece, alongside features. This is exactly what my next post will be about.

This post was the forth post in a series about how to get sponsorship for a large scale refactoring.

5 Mistakes Badass Developers Never Do

Here is a one sentence summary of my previous post.

Badass developers negotiate large scale refactorings with the business better.

Unfortunately, not all of us are sitting next to a true badass developer … Hopefully, we can all become one ! Depending on our track record, it’s going to be more or less difficult, but with time and the good attitude, we can all do it. Becoming a badass developer is all about trustworthiness.

This post is the third in a series about how to get sponsorship for a large scale refactoring. If you haven’t, I recommend you to start from the beginning.

The first thing to become trustworthy is to avoid things that kill trust. Sounds obvious, but it’s very easy to forget. Here are 5 examples of trust killers you should never do if you want to become a badass developer.

Drawing of a hurt finger after someone made a mistake with a hammer. Badass developer don't do this kind of mistakes !

Resume Driven Development

We should pick the best tools for the job. The best tools are often a bit old, precisely because they’ve been battle tested in production for a while ! That’s exactly the kind of technologies you want your business to rely on.

To keep his skills up to date, a badass developer will not add a funky new tech in the production code. He would rather negotiate slack time with the business. He might also openly take his Friday afternoons to experiment the latest techs ! He would simply explain that it’s to avoid polluting the production system.

💡 A badass developer will not add a funky new tech in the production code.

Over-engineering

Gold plating or over-engineering are similar anti-patterns. A badass developer always keeps the business’s interest in mind. This means he knows how to balance long term design and short term features. A good rule of thumb is to keep a vision in sight, but to get there in baby steps.

Build features with no agreed upon value

Product managers are here to select what should and what should not be in the product. As product experts, they are the ones who know how much a feature is worth. Except (maybe) when we are building tools for others developers, they know better than us. Adding something of dubious value in the product is bad in two ways. 

  • First, it takes some time to build, time that we could use to build valuable features instead. Remember : Real developers ship !
  • Second, it creates unnecessary code to maintain. In the worst case, it can constraint the architecture. Which might eventually prevent us from adding other more valuable features afterwards.

Hide in a tunnel

We should always be careful of the tunnel effect. Seeing their money vanishing with no visible output makes business people, understandably, creepy. As soon as things become too complicated, a badass developer will raise the alarm. The fact is that he has been in this kind of situation before, and knows how to recognize it. At that point, it’s still possible to discuss the problem with everyone, and adjust the plan.

As an interesting side note, I was at the Paris DDD Meetup last Thursday. We had the chance to welcome Eric Evans as a surprise guest ! When asked what were his worst mistakes, he said something along this line :

💡 Some of my biggest mistakes were not backtracking soon enough a few times as I was drifting in quagmire. Eric Evans

Eric Evans, the father of DDD, a true badass developer, answering questions at the Paris DDD meetup


Let the team down

It’s not only about getting the business’s trust. We must also build trust from our fellow developers. Whenever we break the build and leave, or worse, deploy and leave, that trust is gone for a long while… We should not do that !

There’s more to a badass developer

I’m done with this list of things badass developers don’t do. Avoiding these is only the first step to become a badass developer. In next post, I’ll write about what we need to do if we want to build strong trust with the business.

This post is the third post in a series about how to get sponsorship for a large scale refactoring.

Why We Need Badass Developers to Perform Large Scale Refactorings

My last post was about the challenge for dev teams to get sponsorship for large scale refactorings. I listed two main reasons :

  1. The business doubts developers to have their interests in mind
  2. They are also not aware of the cost of the current technical debt

This post (and the next) will be about how to gain the business’s trust. This is exactly where badass developers can help. Let me start with a story.

Drawing of 2 hands of a badass developer over his keyboard, with ">badass<" tatooed on his fingers

Back in 2002, at my first job, I started to read Refactoring, Improving the Design of Existing code. That’s where I read about unit testing. I found it so great that I made a demo to other programmers. Being the junior dev in the team, my co-workers reaction was something between “Meh” and “Maybe”. Fortunately, a more experienced and respected developer gave it a try. A few weeks after my demo, he announced to the team that unit testing worked great on new code. This time, people showed no questioning about his opinion : he if said so, it must have been true. Even business people blessed the practice !

The "Refactoring, improving the design of existing code" cover. Badass developers know how to perform large scale refactoring

I had given a good live coding demo, but it was this respected developer’s opinion that won the point. To convince business people of sponsoring a large scale refactoring, we need their trust. That’s why we need badass developers around.

💡 I had given a good live coding demo, but it was this respected developer’s opinion that won the point.

What is a badass developer

Badass Developer's fist with a ring "I am badass"

By Brooke LarkCC0, via Wikimedia Commons

Badass developers are first of all people who are credible to the business. This usually implies a track record of delivering features and refactorings. Badass developers understand the business constraints. That’s why they learned how to deliver refactorings alongside features. They also need to be responsible and bold enough to stand ground in front of the business. Finally, badass developers are able to train others.

💡 Badass developers are first of all people who are credible to the business

Unfortunately, there are not so many badass developers in the industry … It has a youngster bias, and tends to push experienced developers to other activities. As if 10 years of systems design was less valuable than knowing the latest language !

Learn more about Badasss developers

I tried to find other words before resorting to ‘Badass’. Unfortunately, I could find none that got the point so clearly. Uncle bob calls them ‘software professionals’ in The Clean Coder. Professionalism is not specific enough to me. Adam Nowak also calls them ‘responsible developers’ in a blog post.  That does not convey the idea that, sometimes devs need to stand guard in front of the business.

The clean coder book cover. Clean coder looks like a form of badass developer

These concepts, though, are very close to my definition of a badass developer. Check by yourself :

To be continued

This was why badass developers matter to the success of large scale refactorings. This was the second post in a series about how to get sponsorship for a large scale refactoring.  In the next post, we’ll look at what we can do to all become Badass developers.

How to Convince Your Business of Sponsoring a Large Scale Refactoring

Whenever I present or suggest a good practice to dev teams, I often get the same remark. Here is how it goes :

  • That’s a great idea and we would love to do this, but our code is in such a mess that we cannot !

  • Maybe you should start doing more refactoring then !

  • We would like to, but we don’t have the time. We are fire fighting all the time.

It’s a bit like the old adage of the lumberjack that is too busy to cut wood to sharpen his axe… The sad part here, is that most of the time, developers know they would be a lot faster if they could clean up their code. Unfortunately, they are usually not given the time.

How do we end up in this silly situation ?

Drawing of a '5 whys' mind map explaining why it is difficult to get sponsorship for a large scale refactoring

Only developers see the bad code

As I’ve already been joking about, code is invisible. Mess in the code even more so, especially to people who don’t code. The code could look like that and no one would notice.

Inside of a kitchen from someone suffering from Diogenes syndrome

Par Un TouristePhotographie personnelle, CC BY-SA 3.0, Lien

If someone put his own office in that state, he would get fired, but not for the source code. The good side is that we, developers, are safe, we can continue to wreak chaos without fear ! That’s pretty weird when we think that this is what we ship to customers …

💡 Is Diogenes syndrome for source code a recognized pathology ?

Business might also not see bad code because that’s the only thing they’re used to ! Maybe they’ve always been working in dysfunctional organizations that systematically create crappy code. Slow teams, late deliveries and fire fighting might be business as usual for them. From this point of view, trying to improve code is a pure waste of time and energy. The same goes for large scale refactorings.

The worse part of all this is that if devs don’t have the time to keep their code clean, it will only get worse. This will reinforce the view that software delivery is slow and that there is nothing to do about it !

Business has been burnt in the past !

Bad experiences are another reason why business is unwilling to sponsor refactoring. Did someone sell them an unrealistic productivity boost that turned in a never-ending tunnel project ? Badly managed large scale refactorings deliver late, create no value, and a lot of bugs. At one company I worked for, business gave devs 1 full year (!) to clean up the code … We took 2 !! Meanwhile, the CEO had to dilute the stocks a bit to keep the boat afloat ! I’d think twice before giving this kind of mandate myself.

Performing a large scale refactoring is not easy, and involves specific skills. These skills are about refactoring in baby steps, alongside feature delivery.

Usually, people acquire these skills through hard won experience … Unfortunately for us, our industry is not very nice to experienced engineers … It’s a lot easier to hire a fresh grad who knows the latest javascript framework than a 2 decades engineer. (Who, BTW, could learn this framework in 2 weeks …) It’s also a lot harder for the junior developer to succeed in negotiating a refactoring.

Again the twist of fate is that junior engineers are a lot more likely to start a submarine latest-framework.js rewrite supposed to solve all maintenance issues … which will only make things worse.

Overestimate, only as last resort

A quick fix is to systematically overestimate to get enough time to refactor. As any other ‘submarine’ initiative, I would recommend it only in last resort, after you’ve tried every other possible technique … and just before you quit.

Hiding things to the business people kills trust and hides problems. Trust and collaboration is what you need to get the business to sponsor large scale refactorings ! Plus, if ever you mess up (as submarine initiative often do) you’ll be the only one to blame …

That said, ‘overestimating’ so that you can write clean code is ok. It’s not overestimating, it’s estimating to do a good job.

💡 We should never ask the permission to do a good job. (Doc Norton)

To be continued

You might wonder what these other techniques are ! That’s exactly what I’ll go through with the next posts. This was the first one in a series about how to get sponsorship for a large scale refactoring. The series will cover topics like :

  1. How to convince your business of sponsoring a large scale refactoring
  2. Why we need Badass developers to perform large scale refactorings
  3. 5 mistakes badass developers never do
  4. Principles That Will Make You Become a Badass Developer
  5. Incremental Software Development for Large Scale Refactoring
  6. Techniques to refactor in parallel with features
  7. Nothing convinces business people like money

How to Avoid Unnecessary Meetings (a Takeaway From Devoxx France 2018)

I had the chance to attend Devoxx France this year in Paris. Here is the most important lesson I learned :

How to avoid unnecessary meetings with asynchronous decision making

Bertrand Delacretaz, a member of the Apache foundation. He gave a great talk about how the open source community handles decision taking. Open source developers are often all over the world, often in different timezones. Meetings are not an option for them. Still, they manage to make great decisions !

Drawing of a decision hammer

Even if you don’t work remotely, avoiding unnecessary meetings is always a great thing !

  1. You’ll have more time to do productive and interesting stuff
  2. You’ll avoid interruptions and be even more productive
  3. If you are an introvert, it’s going to be easier to contribute to the decision
  4. As people have more time to think through the decision, the result is usually better

For a full walkthrough, I encourage you to watch the talk in full length. If you don’t speak french, an english version is available here. Finally, slides are also available in french and english.

💡 Even if you don’t work remotely, avoiding unnecessary meetings is always a great thing !

Crash course

For the hasty folks among you, here is a summary. The decision making follows 4 stages :

  1. Open discussion and brainstorming. People discuss openly and suggest ideas in a free form manner.
  2. Emergence of options. After enough discussion, a few options will start to make more sense than others.
  3. Coming to a consensus. Someone will draft a formal proposal. People will discuss and amend this proposal until they reach consensus. Consensus is not unanimity !
  4. Decision. After consensus, the benevolent decision owner validates the decision once and for all.

Until the decision is taken, the process can move forward but also backward.

Tooling

We need only two tools to make this possible :

  1. For discussion, brainstorming and emergence of options, use a very open and chatty tool. The speaker called this a “shared asynchronous communication channel”. This can be an online chat, a mailing list or Github issues (ex). It could even be a real life whiteboard if you all had access to it.
  2. From drafting the proposal to the end, prefer a structured and chronological tool. The speaker suggests using a “shared case management tool”. Draft the proposal in this tool, and use comments to log the latest steps of the decision taking. He had examples using Jira issues (ex) or Github pull requests (ex). To confirm the decision, close the case. The tool will record which version of the decision was exactly taken.

Architecture Decision Record

Drawing of an Architecture Decision Record which work great with asynchronous decision making

ADR is the practice of documenting architecture decisions. It makes sure we remember why we took a decision. This can be very useful to know how to deal with the existing software. A widespread practice for ADRs is to use simple text files in git. There are even tools for that. This looks like a perfect fit for decision making using git pull requests ! I’ll write a post about that when I get the chance to try.

💡 Git pull requests based asynchronous decision making is a perfect fit for Architecture Decision Records.

Currently experimenting

I am currently trying this whole decision making technique at work. We are still in the brainstorming phase. We are using our internal chat app for that. Options are starting to emerge, but we did not move to the consensus part yet. I’ll write a return on experience post when we reach the end.

A Coding Dojo Exercises Plan Towards Refactoring Legacy Code

My current job at work is technical coach. I’m available for teams that need help to adopt incremental coding practices.

Problems with refactoring legacy code

A few months ago, a team which was struggling with a lot of legacy code asked for help. As you might know if you read my blog, I’m a big fan of Test Driven Development (TDD) because it has made my life as a developer so much more easy. I’m so used to TDD now, that even if I don’t have tests yet (as is the case when refactoring legacy code), TDD helps me :

  • To stick to baby steps which are a lot less likely to fail than larges changes.
  • Write testable code. I know what testable code looks like, and when refactoring, I’ll try to change it towards that.

That’s why we started to run regular, all team, coding dojo randoris. It was nice for the team dynamics, and the people where learning a lot of technical skills. I also got the feedback that they where not able to apply this directly on their day to day job though. After a bit more discussion, I understood that they did not know where this was going, what to expect, and when !

💡 Test Driven Development also teaches you what testable code looks like.

The coding dojo exercices

It turned out that a coding dojo exercises plan was enough to answer their questions. This is what it looks like.

Drawing

An illustrated Coding Dojo Exercises plan leading to the mastery of Legacy Code Refactoring

Mind Map

Here is another, more concrete, version, with sample names of katas we can find online.

An mind map of Coding Dojo Exercises plan leading to the mastery of Legacy Code Refactoring

Text

It starts with simple greenfield katas :

It goes on to intermediate katas, where we can use TDD to do design :

From then on, it’s possible to tackle advanced katas and styles :

All this opens the gate to legacy code refactoring katas :

At that point, the team can mob to refactor production code :

  • Real life, static analysis issue, mob programming session
  • Real life, code smell, mob programming session
  • Real life, larger mob Refactoring

What changed in practice ?

We wanted to split the teamwork and the coding dojos exercises. The team is now doing mob programming sessions on their usual stories twice a week (I’ll blog about that someday). But also doing regular coding dojos exercises in pairs.

Even if they did not go through all the TDD katas yet, mobbing on real stories helps the team to take on legacy code.

Given enough eyeballs, all bugs are shallow. Linus’s Law

Working in pairs on the code katas allows them to be more engaged in the exercises. In the end, it brings faster learning.

💡 A mix of Coding Dojos in pairs and Mob Programming sessions is a good way to teach TDD in a Legacy Code context.

When Is Testing Using Mocks Still a Good Idea ?

In the previous 7 articles of this series, I’ve tried my best get rid of mocks. I’m pretty sure that using these techniques will get you a long way out of mock hell. Excessive mocking leads to unmaintainable tests. Unmaintainable tests lead to low coverage. Low coverage ultimately leads to legacy code. If you haven’t already, I encourage you to start reading from the beginning.

One question remains though : Is it realistic to get rid of all mocks ? An even better question would be : Are mocks always bad ? Are there situations when mocking is the best choice ?

When mocking still makes sense

Let’s to through a few examples.

Testing a generic wrapper

A few years ago, I had to write a service for an enterprise system. As any service, I had to ensure that it was returning nice errors. We decided to capture and wrap all errors from a few ‘gate’ points in the code. We built a generic wrapper that did only delegation plus exception wrapping. In this case, it made a lot more sense to test this with a mocking framework.

1
2
3
4
5
6
7
8
9
10
11
context ServiceErrorWrapper do

 specify 'converts all kinds of exceptions' do
   failing_object = object_double("Failing object")
   allow(failing_object).to receive(:long_computation).and_raise(Exception.new("Something terrible happened"))

   expect{ ServiceErrorWrapper.new(failing_object).long_computation }.to raise_error(ServiceError).with_message("Something terrible happened")
 end

 # ...
end

Not only did we reuse the wrapper many times in my service. We also ended up using it in other services as well !

Injecting a hand written in-memory fake

As you might have noticed, in the previous article, I recommended to use an in-memory fake instead of mocks. By nature, an in-memory fake is a kind of mock. Even if it is not defined by a mocking framework. (I actually think that by making mocking so easy, mocking frameworks often do more harm than good.)

💡 By making mocking so easy, mocking frameworks often do more harm than good.

Still, I used const_stub(...) to inject the in-memory fake.

1
2
3
4
5
config.before(:each) do  

  stub_const("TwitterClient::Client", FakeTwitterClient.new)  

end  

I did this for 2 reasons :

  • Production code can continue to use a straightforward constant
  • I don’t risk forgetting to remove the mock at the end of its lifecycle, the framework does this for me
  • As I’m injecting the same fake for all tests, there is not much risk of test conflict (for the moment)

Testing a cache

The “raison d’être” of a cache is to avoid doing something twice. It should also return the same results as if it was not there. This is by nature almost impossible to test with state based assertions. Mock frameworks are great for this situation though. Here is an example :

1
2
3
4
5
6
7
8
context "UsersController" do
 it 'caches users' do
   expect(User).to receive(:load).once.and_return(User.new(name: "Joe"))

   controller.login('Joe', 'secret')
   controller.login('Joe', 'secret')
 end
end

The assertion could not be more explicit, we are checking that the expensive load was only done once.

Legacy code

Michael C.Feathers explains that testing using mocks is a key practice in "Working Effectively with Legacy Code"

In Working Effectively with Legacy Code Michael Feathers explains how to exploit “seams” in the code to put it under test. Mocking is straightforward way to inject behavior through a seam.

Mocking is a pretty good starting point but we need to be careful and keep a few things in mind. Legacy or not, we must not forget that too many mocks will make tests unmaintainable !

  • It’s a good idea to refer to a target design or architecture blueprint to know where to inject mocks. (I’ll write a post about this one day). This increases the chances to replace them with an in-memory fake later down the road.
  • Plan to replace the mocks with a better design as soon as possible.

It depends …

As with anything in software, there is no absolute rule about mocking. Even if I prefer not to 99% of the time, there are situation when testing using mocks is the thing to do. Knowing the risks, it’s up to you to decide !

If using a mock, prefer spy / proxies

Spies and proxies make testing using mocks less intrusive

As I explained in previous posts, mocks duplicate behavior. If we could use mocks without duplicating behavior, they would do less harm.

It turns out there is a flavor of mocks for that : spies and overlooked proxies. Proxies do the real thing but also record the calls and return values. It’s as non-intrusive as mocks can be.

💡 Proxy mocks are as unintrusive as mocks can be.

For example, here is how our cache test would look like using a proxy :

1
2
3
4
5
6
7
8
9
10
context "UsersController" do
 it 'caches users' do
   allow(User).to receive(:load).and_call_original

   controller.login('Joe', 'secret')
   controller.login('Joe', 'secret')

   expect(User).to have_received(:load).once
 end
end

It’s more verbose, but simpler. Most mock frameworks provide some form of spy or proxies. A few years ago, I also wrote rspecproxies, a wrapper on top of rspec to make this easier.

This is the end

This was the 8th and last post in a series about how to avoid mocks. Before closing here is a list of other references about the topic.

Get Rid of Mock Maintenance With Full Fledged In-memory Fakes

Last week’s post was about how hexagonal architecture results in fast, mock-free tests around your core domain. Unfortunately, that does not remove all mocks, yet it groups them in the same, less critical, zone. In last week’s code sample, this was the controller. I concluded that at least, this was easier to manage. Let’s see how.

Hand written 'In-memory fake' with memory replaced by a RAM board

This is the 7th post in a series about avoiding mocks. If you haven’t, you might start from the beginning.

Mock concentration

Let’s get back to the last post’s code sample. As a reminder, it’s a very basic TODO app built on Rails. I extracted the domain part, the tasks, in a core domain area. This allowed to push all mocks out of this section. A consequence though, is that all mocks gathered in the controller test. Here is the controller code :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
require 'core/task'
require 'infrastructure/task_repo'

class TasksController < ApplicationController
 before_action :set_task, only: [:show, :edit, :update, :destroy]

 # GET /tasks  
 def index
   @tasks = Infrastructure::TaskRepo.all
 end

 # GET /tasks/1  
 def show
 end

 # GET /tasks/new  
 def new
   @task = Core::Task.new
 end

 # GET /tasks/1/edit  
 def edit
 end

 # POST /tasks  
 def create
   begin
     @task = Core::Task.new(task_params)
     Infrastructure::TaskRepo.save(@task)

     redirect_to task_url(@task.db_id), notice: 'Task was successfully created.'

   rescue ArgumentError
     render :new
   end
 end

 # PATCH/PUT /tasks/1  
 def update
   begin
     @task.update(task_params)
     Infrastructure::TaskRepo.save(@task)

     redirect_to task_url(@task.db_id), notice: 'Task was successfully updated.'

   rescue ArgumentError
     render :edit
   end
 end

 # DELETE /tasks/1  
 def destroy
   Infrastructure::TaskRepo.delete(@task)
   redirect_to tasks_url, notice: 'Task was successfully destroyed.'
 end

 private
   def set_task
     @task = Infrastructure::TaskRepo.load(params[:id])
     @task.notify_when_done do |task|
       TwitterClient::Client.update(task.description)
     end
   end

   # Never trust parameters from the scary internet, only allow the white list through.  
   def task_params
     params.permit(:description, :done)
   end
end

The controller is now dealing both with the Twitter connection and the database. This is visible in the controller test :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
require 'rails_helper'

RSpec.describe TasksController, type: :controller do

 before :each do
   allow(TwitterClient::Client).to receive(:update)
 end

  # ...  

 describe "PUT #update" do
   context "with valid params" do
     let(:new_attributes) {
       {done: true}
     }

     it "updates the requested task" do
       task = Task.create! valid_attributes
       put :update, params: new_attributes.merge(id: task.to_param)
       task.reload
       expect(task).to be_done
     end

     it "tweets about completed tasks" do
       task = Task.create! valid_attributes

       expect(TwitterClient::Client).to receive(:update).with(task.description)

       put :update, params: {id: task.to_param, done: true}
     end

     it "redirects to the task" do
       task = Task.create! valid_attributes
       put :update, params: valid_attributes.merge(id: task.to_param)
       expect(response).to redirect_to(task_url(task.id))
     end
   end

   # ... 

  end
end

We need to stub out the twitter API for most tests. We are also still using a mock to verify that the tweet is sent. Finally, as we can see from the test execution times, we are still using the database in some tests.

Screen capture of the tests execution time

If the project grew large this would become an issue. Sadly, mocking is often the fix people jump on …

💡 Mocking is the unfortunate quick fix to slow tests.

From a mocking point of view, our current controller test can seem worse than before ! There’s something pretty effective we can do though !

In memory fakes

Instead of stubbing and mocking in every test, let’s write a full fledged in-memory fake that does the job we need. We could then install it once and for all, and forget about it !

Actually, this is nothing new. This is exactly what Rails provides out of the box with ActionMailer::Base.delivery_method = :test.

Here’s how we could do the same thing for our Twitter Client.

spec/rails_helper.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
class FakeTwitterClient
 def initialize
   @tweets = []
 end

 attr_accessor :tweets

 def update(message)
   @tweets.push(message)
 end
end

RSpec.configure do |config|

  # ...  
 config.before(:each) do
   stub_const("TwitterClient::Client", FakeTwitterClient.new)
 end
end
spec/controllers/tasks_controller_spec.rb
1
2
3
4
5
6
7
it "tweets about completed tasks" do
 task = Task.create! valid_attributes

 put :update, params: {id: task.to_param, done: true}

 expect(TwitterClient::Client.tweets).to include(task.description)
end

Simple isn’t it ?

Wait a sec …

There’s a catch though … How do we make sure that this fake is behaving the same way as the real thing ?

Let’s run the same tests on both ! We could mimic the twitter API in our fake, but that might not be a great idea. Do you remember the moto “Always wrap your 3rd parties” ? It takes all its meaning here, for 2 reasons.

The first is to make faking easier. We can build a minimal wrapper API that is just enough for our use. By keeping this interface small, we’ll make it a lot easier to fake.

The second reason is that we can write real integration tests on the 3rd party through this wrapper. They’d look like ordinary unit tests, except that they’d end up calling the real 3rd party in a sandbox. They are usually pretty slow, but as 3rd parties don’t change everyday, that’s ok. We can ensure up-front that integration will go well. As a bonus, we can be very fast to detect and contain changes to online services. (I’m looking at you Scrappers!)

Here is what it would look like for our Twitter client :

lib/infrastructure/twitter_client.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
class FakeTwitterClient
 def initialize
   @tweets = []
 end

 attr_accessor :tweets

 def tweet(message)
   @tweets.push(message)
 end

 def search_tweets(text)
   @tweets.select {|tweet| tweet.include?(text) }
 end
end

class RealTwitterClient
 def initialize(&block)
   @client = Twitter::REST::Client.new(&block)
 end

 def tweet(message)
   @client.update(message)
 end

 def search_tweets(text)
   @client.search("from:test_user #{text}")
 end
end

As you can see, we renamed update to tweet in the wrapper. We’d have to update the calls accordingly. Let’s look at the tests.

spec/lib/Infrastructure/twitter_client_spec.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
require 'rails_helper'
require 'infrastructure/twitter_client'
require 'securerandom'

RSpec.shared_examples "a twitter client" do |new_client_instance|
 let(:client) { new_client_instance }
 it "sends tweets" do
   token = SecureRandom.uuid
   message = "Philippe was here #{token}"
   client.tweet(message)

   expect(client.search_tweets(token)).to include(message)
 end
end

context FakeTwitterClient do
 it_behaves_like "a twitter client", FakeTwitterClient.new
end

context RealTwitterClient, integration: true, speed: :slow do
 it_behaves_like "a twitter client", (RealTwitterClient.new do |config|
   config.consumer_key        = "TEST_CONSUMER_KEY"
   config.consumer_secret     = "TEST_CONSUMER_SECRET"
   config.access_token        = "TEST_ACCESS_TOKEN"
   config.access_token_secret = "TEST_ACCESS_SECRET"
 end)
end

We had to add a search method to our interface for the sake of testing. This should remain “For testing only”. We’d also adapt the controller test to use this search_tweets method.

Let’s look at where we stand now. We’re injecting each mock only once. Tests are fast yet straightforward, almost as if they were testing the real thing. Doing so, we’ve split our system in cohesive parts and we’ve wrapped our 3rd parties. We’ve actually done a lot more than removing mocks ! Mocking really is a design smell.

💡 Merciless mock hunting will improve the design of your system !

Last word about implementation

Sometimes, this 3rd party wrapper can become pretty complicated. Try to reuse as much of it as possible between the real and the fake. For example, an ORM, like ActiveRecord for example, is a wrapper around the database. Reimplementing a fake ORM would be real challenge. We’re far better plugin it on top of SQLite instead !

References

Smart people have already spoken and written about this subject. If you want to learn more, I recommend that you have a look at Aslak Hellesøy’s Testable Architecture talk. James Shore, the author of The Art of Agile Development, also wrote a pattern language called Testing Without Mock.

Next week

This was the 7th blog post in a series about how to avoid mocks. Hopefully, I’m reaching the end ! Next week’s post should be the last in series, and deal with a few remaining points. What to do when you really need a mock ? What about mocking and legacy code ?

Avoid Mocks and Test Your Core Domain Faster With Hexagonal Architecture

As I’ve written in my last few posts, we can get a long way to avoid mocks with small scale coding best practices. Unfortunately, when systems reach a certain size, we need something at architecture scale.

This is the 6th post of a series about avoiding mocks. If you haven’t, you can start by the beginning.

A drawing of a hexagon-shaped building

Why do we end up with mocks in large systems ?

A few years ago, I joined a team working in a legacy system. We wanted to apply TDD and refactoring. As expected, adding tests legacy code proved a real challenge. With a lot of effort we could manage to add a few. Unfortunately, this did not seem to have any positive effect on our maintainability ! The tests we were writing all involved a lot of mocking. The system was such a large mass of spaghetti code that there was no clear place to mock. We were actually mocking where it seemed the easiest on a test by test basis. We were making progress at small scale, but the big picture was not improving at all !

Large systems are beasts with many faces. They  involve a lot of IOs. They write and read data from the disk and databases. They call 3rd parties and remote services.

As we test these large systems, we’ll need to stub out these IOs. Even if the tests are fast enough, we usually don’t want to call external services for real. Most of the time though, tests are slow. That’s 2 reasons why end up adding some mocks.

Here comes the nasty part. These large systems are so complex that we, developers, don’t have the full picture. When we test, we tend to mock at different places, depending on our knowledge. This is bad for maintenance. Mocks duplicate production code behavior. When many different mocks are in place to isolate an external dependency, we end up with ‘n’ versions of the code. That’s a nightmare to refactor !

💡 When many different mocks are in place to isolate an external dependency, we end up with ‘n’ versions of the code !

Hexagonal architecture to the rescue

Alistair Cockburn coined the term. The idea is pretty simple :  isolate a piece of code from all dependencies. This is particularly useful for the core functional areas. With this in place, it becomes straightforward (and fast) to test the core domain logic.

To main techniques to isolate a piece of code from any dependency are :

It’s also possible to split a system in many ‘hexagons’ and glue them together with adapters at startup. If you want to learn more on this style of architecture, have a look into the Domain Driven Design lore. This community has been building systems this way for years now.

Enough talk, show me the code !

This post was the occasion to try to inject a Hexagonal Architecture and a dash of DDD in a Rails application. There’s one caveat though : DDD shines on complex systems. Unfortunately, large and complex systems make very poor didactic examples. The following code highlights the gains about mocking. We would not use DDD for such a small app in real life.

The starting point

I chose a simple TODO app. I started by generating a scaffold for a Task with a description and a done/not-done status. As third party interaction, completing a task sends an automatic tweet. Here is the only specific code I wrote on top of the Rails scaffold :

app/models/task.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
class Task < ApplicationRecord
  include ActiveModel::Dirty

  validates :description, presence: true

  before_save :tweet_if_done

  private
  def tweet_if_done
    if done_changed?
      TwitterClient::Client.update(self.description)
    end
  end
end

Thanks Jason Charnes for the change attribute technique.

spec/models/task_spec.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
require 'rails_helper'

RSpec.describe Task, type: :model do

  it "is valid with all attributes set" do
    expect(Task.create(description: "Finish presentation", done: false)).to be_valid
  end

  it "requires a description" do
    expect(Task.create(description: nil, done: false)).to be_invalid
    expect(Task.create(description: "", done: false)).to be_invalid
  end

  it "tweets when a task is finished" do
    task = Task.create(description: "Wash the car", done: false)

    expect(TwitterClient::Client).to receive(:update).with("Wash the car")

    task.done = true
    task.save
  end
end

This is pretty simple and to the point !

5 years later

Now let’s imagine that the app grew to tens of thousands of lines. We added a lot of features to the app, which transformed the TODO domain into a very complex thing. Now suppose that, for the sake of maintenance, we want to isolate the domain logic into its own hexagon. Unlike traditional Rails ActiveRecords, we want to make it independent from the database. We also want it to be independent from the Twitter API.

Here is what the code might look like.

lib/core/task.rb

First, we have a core task class, independent from anything else. The Core module is our hexagon.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
module Core
  class Task

    attr_reader :description
    attr_accessor :db_id

    def initialize(attributes = {})
      @description= "What do you need to do ?"
      @done = false
      @done_subscribers = []

      self.update(attributes)
    end

    def done?
      @done
    end

    def mark_as_done
      @done = true
      @done_subscribers.each {|proc| proc.call(self) }
    end

    def update(attributes={})
      self.description= attributes[:description] unless attributes[:description].nil?
      self.mark_as_done if attributes[:done]
    end

    def notify_when_done(&proc)
      @done_subscribers.push(proc)
    end

    def description=(desc)
      raise ArgumentError.new("Task description cannot be blank") if desc.blank?

      @description = desc
    end
  end
end

As we can see, it contains only domain logic and nothing else.

# spec/lib/core/task_spec.rb

Here is the corresponding test, fast, mock-free and independent from the database and any external system.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
require 'rails_helper'
require 'core/task'

context 'Task' do

  let(:task) { Core::Task.new}

  specify 'is not done by default' do
    expect(task).not_to be_done
  end

  specify 'comes with a default description' do
    expect(task.description).not_to be_blank
  end

  specify 'it can be initialized from a hash' do
    task = Core::Task.new(description: "Old description", done: true)

    expect(task.description).to eq("Old description")
    expect(task).to be_done
  end

  specify 'can have a custom description' do
    task.description= "Clean up the house"
    expect(task.description).to eq("Clean up the house")
  end

  specify 'forbids empty descriptions' do
    expect{task.description = nil }.to raise_error(ArgumentError)
    expect{task.description = "" }.to raise_error(ArgumentError)
  end

  specify 'can be done' do
    task.mark_as_done
    expect(task).to be_done
  end

  specify 'publishes when done' do
    done_task = nil
    task.notify_when_done {|t| done_task = t}

    task.mark_as_done

    expect(done_task).to be(task)
  end

  specify 'can be updated with a hash' do
    task.update(description: "New description", done: true)

    expect(task.description).to eq("New description")
    expect(task).to be_done
  end

  specify 'has no DB id by default' do
    expect(task.db_id).to be_nil
  end
end
# lib/infrastructure/task_repo.rb

To read and save with the database, we now go through an adapter. This is not considered to be part of our core domain.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
module Infrastructure
  class TaskRepo

    def self.all
      Task.all.map do |db_task|
        from_db(db_task)
      end
    end

    def self.load(db_id)
      from_db(Task.find(db_id))
    end

    def self.save(task)
      if task.db_id.nil?
        db_task = Task.create!(to_db_attributes(task))
        task.db_id = db_task.id
      else
        db_task = Task.find(task.db_id)
        db_task.update!(to_db_attributes(task))
      end
      task
    end

    def self.delete(task)
      unless task.db_id.nil?
        db_task = Task.find(task.db_id)
        db_task.destroy!
        task.db_id = nil
      end
    end

    private

    def self.to_db_attributes(task)
      {description: task.description, done: task.done?}
    end

    def self.from_db(db_task)
      result = Core::Task.new
      result.db_id = db_task.id
      result.description = db_task.description
      result.mark_as_done if db_task.done?
      result
    end

  end
end
# app/controllers/tasks_controller.rb

Finally, all the pieces interact together in the controller. This controller basically does what the previous version was, it’s just using different classes. Obviously, we’ll need to adapt the views and the tests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
require 'core/task'
require 'infrastructure/task_repo'

class TasksController < ApplicationController
  before_action :set_task, only: [:show, :edit, :update, :destroy]

  # GET /tasks
  def index
    @tasks = Infrastructure::TaskRepo.all
  end

  # GET /tasks/1
  def show
  end

  # GET /tasks/new
  def new
    @task = Core::Task.new
  end

  # GET /tasks/1/edit
  def edit
  end

  # POST /tasks
  def create
    begin
      @task = Core::Task.new(task_params)
      Infrastructure::TaskRepo.save(@task)

      redirect_to task_url(@task.db_id), notice: 'Task was successfully created.'

    rescue ArgumentError
      render :new
    end
  end

  # PATCH/PUT /tasks/1
  def update
    begin
      @task.update(task_params)
      Infrastructure::TaskRepo.save(@task)

      redirect_to task_url(@task.db_id), notice: 'Task was successfully updated.'

    rescue ArgumentError
      render :edit
    end
  end

  # DELETE /tasks/1
  def destroy
    Infrastructure::TaskRepo.delete(@task)
    redirect_to tasks_url, notice: 'Task was successfully destroyed.'
  end

  private
    def set_task
      @task = Infrastructure::TaskRepo.load(params[:id])
      @task.notify_when_done do |task|
        TwitterClient::Client.update(task.description)
      end
    end

    # Never trust parameters from the scary internet, only allow the white list through.
    def task_params
      params.permit(:description, :done)
    end
end

The main gain here is that our core domain, our most valuable asset is now easy to test without mocks. This means that we are able to write and execute fast tests for this area of the code. This puts us in a great position to increase our competitive advantage in our core business !

💡 By keeping your tests around your core domain fast, Hexagonal Architecture increases your competitive advantage.

As you can see, we are now wiring everything together at the controller level. We could later build a facade to isolate the controller from the inside of our domain. A presenter might do, but it seemed over-engineered, even in this made up example. (I’ll post something about that some day)

Next post

As we can deduce from the controller code above, we still have to use fakes or mocks when testing the controller. The good thing though is that this is now more local which already makes mocking less of an issue. If a mock is used in less tests, it’s easier to use the same mock everywhere ! This is a great opportunity for simplifying test setup, as we’ll see in the next post about in-memory fakes.

How Custom Assertion Matchers Will Keep Mocks Away

I cannot write a series about avoiding mocks without mentioning Custom Assertion Matchers. If you don’t know what custom assertions are, here is pseudo code that uses a custom assertion :

1
assert.that(actual, VerifiesMyCustomAssertion(withCustomProperties))

For more details, have a look at these examples for your preferred language : Java, Ruby or Javascript.

A drawing of a box of matches, branded 'Matchers' on top

That custom assertion matchers have an effect on mock usage might seem puzzling at first. Let me explain. Us, mere human developers, get lured into mocking when tests become too complicated. By keeping the tests simpler, Custom Assertion Matchers help use to avoid mocks. It’s a bit like why test data builders keep mocks at bay.

💡 We get lured into mocking when tests become too complicated

I already blogged about the benefits of Custom Assertion Matchers. Here I’m going to dive in their advantages against mocking.

This is the fifth post in a series about how to avoid mocks. If you haven’t yet, I recommend you to start from the beginning.

Why would we end up with mocks when we don’t have matchers ?

Let’s walkthrough a small story. Suppose we are building an e-commerce website. When someone passes an order, we want to notify the analytics service. Here is some very simple code for that.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
class AnalyticsService

 def initialize
   @items = []
 end

 attr_reader :items

 def order_passed(customer, cart)
   cart.each do |item|
     @items.push(customer: customer, item: item)
   end
 end
end

class Order
 def initialize(customer, cart, analytics)
   @customer = customer
   @cart = cart
   @analytics = analytics
 end

 def pass
   # launch order processing and expedition  

   @analytics.order_passed(@customer, @cart)
 end

end

describe 'Order' do

 it "notifies analytics service about passed orders" do
   cart = ["Pasta","Tomatoes"]
   analytics = AnalyticsService.new
   order = Order.new("Philippe", cart, analytics)

   order.pass

   expect(analytics.items).to include(customer: "Philippe", item: "Pasta")
   expect(analytics.items).to include(customer: "Philippe", item: "Tomatoes")
 end
end

Let’s focus on the tests a bit. We first notice that the verification section is large and difficult to understand.  Looking in more details, it knows too much about the internals of AnalyticsService. We had to make the items accessor public just for the sake of testing. The test even knows how the items are stored in a list of hashes. If we were to refactor this representation, we would have to change the tests as well.

We could argue that responsibility-wise, our test should only focus on Order. It makes sense for the test to use a mock to verify that the Order calls AnalyticsService as expected. Let’s see what this would look like.

1
2
3
4
5
6
7
8
9
it "notifies analytics service about passed orders" do
 cart = ["Pasta","Tomatoes"]
 analytics = AnalyticsService.new
 order = Order.new("Philippe", cart, analytics)

 expect(analytics).to receive(:order_passed).with("Philippe", cart)

 order.pass
end

Sure, the test code is simpler. It’s also better according to good design principles. The only glitch is that we now have a mock in place with all the problems I described before.

This might not (yet) be a problem in our example but, for example, the mock ‘cuts’ the execution of the program. Suppose that someday, the Order starts expecting something from the AnalyticsService. We’d then need to ‘simulate’ the real behavior in our mock. This would make the test very hard to maintain.

Matchers to the rescue

Let’s see how a matcher could help us here. The idea is to improve on the first ‘state checking’ solution to make it better than the mock one. We’ll extract and isolate all the state checking code in a custom matcher. By factorizing the code in a single matcher, we’ll reduce duplication. The matcher remains too intimate with the object, but as it is now unique and well named, it’s less of a problem. Plus, as always with matchers, we improved readability.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
RSpec::Matchers.define :have_been_notified_of_order do |customer, cart|
 match do |analytics|
   cart.each do |item|
     return false unless analytics.items.include?(customer: customer, item: item)
   end
   true
 end
end

describe 'Order' do
 it "notifies analytics service about passed orders" do
   cart = ["Pasta","Tomatoes"]
   analytics = AnalyticsService.new
   order = Order.new("Philippe", cart, analytics)

   order.pass

   expect(analytics).to have_been_notified_of_order("Philippe", cart)
 end
end

Here is how we could summarize the pros and cons of each approach :

Assert state Mocks Matchers
👎 duplicated code 👎 duplicates the program behavior ❤️ customizable error messages|
👎 breaks encapsulation ❤️ more readable|
👎 intimacy with the asserted object|
❤️ factorizes the assertion code|

Design improvements

Depending on your situation, you might find further design improvements. In our example, a publish-subscribe pattern might do. A better design is likely to fix the encapsulation problem of the matcher. Here again, the custom assertion matchers will help. In most cases, it will be enough to change the implementation of the matchers only.

💡 Custom assertion matchers make refactoring easier by factorizing test assertions.

Summary of small-scale techniques

I’m done with small scale mock avoiding techniques. To summarize, the first thing to do is to push for more and more immutable value objects. Not only does it help us to avoid mocks, but it will also provides many benefits for production code. Practices like Test Data Builders and Custom Assertion Matchers simplify dealing with Immutable Value Objects in tests. They also help to keep tests small and clean, which is also a great thing against mocks.

Next post

In the following posts, I’ll look into architecture scale techniques to avoid mocks. I’ll start with Hexagonal architecture.