Incremental Software Development Strategies for Large Scale Refactoring #2 : Baby Steps

My previous post was about how to get slots of time in your daily (or weekly) work to do some refactoring. I left my readers with the promise of techniques to fit the refactoring work into these small slots.

Obviously, it won’t be possible to perform any refactoring of any size in this way. With a bit of discipline and know-how though, it is possible to deal with quite a lot by splitting them up.

Baby steps are small increments of working software. The idea is that we test, commit, integrate and even deploy every small code change ! Using baby steps, we can perform large scale refactorings little by little. Refactoring in this way might seem involved, but it’s so much safer that it’s a no brainer once you’ve tried it ! Refactoring in baby steps can be challenging to master though.

Baby footprints. Taking really small baby steps when going through a large scale refactoring is safer

10 years ago, I used to work in a large bank in Paris. I had been dabbling on my own with eXtreme Programming for a few years, when we started a small project. I was to become the informal XP coach. The project was about connecting to an online brokering service. It involved adapting an existing domain library. It went out pretty well. More precisely, we created very few bugs in production, which was very different from the norm at the time. I remember this feedback from the manager :

We managed to move the code from X to Y through a succession of working baby steps ! That’s pretty uncommon ! A manager in 2006

Keep in mind that this was 10 years ago. We had not done anything special except trying to apply eXtreme Programming. Nowadays, as Continuous Integration has become mainstream these skills are becoming more common. That said, we still need to learn how to apply incremental software development to large scale refactoring. This is what I’m going to write about today.

This is the seventh post in a series about how to get sponsorship for large scale refactoring. If you haven’t, I encourage you to start from the beginning.

Team TDD Coding Dojos

Learning to work in baby steps is not as complicated as it might first seem. The safest and easiest way is to setup a team TDD coding dojo. With its Red-Green-Refactor loop TDD sets up a baby steps rhythm. As I’ll explain in my next post, baby steps work best when all the team uses them. That’s another thing the team Coding Dojo helps with.

💡 TDD has a baby steps rhythm baked in.

We can push the learning further. For example, we can use the baby steps constraint during a few coding dojo sessions. With this constraint, we revert the code if tests fail for more than 2 minutes ! Here is a way to go at it :

  1. Setup continuous testing : NCrunch in .Net, Guard in Ruby or Infinitest in Java)
  2. Only use automated refactorings or extremely simple code changes in order to ….
  3. … keep the code compiling all the time …
  4. … and cut the time the tests fail as much as possible

Mikado Method

One way to keep the tests green all the time is to use a slightly different TDD loop, as Nat Pryce suggests :

The red-green-refactor loop of TDD with an extra green arrow from failing test to refactor. This alternate TDD loop illustrates how to take baby steps with the Mikado Method


Here is how it goes. 

  1. Add a new failing test 
  2. If it’s trivial to fix, fix it. We are done
  3. If not, see what’s missing in the code
  4. Comment the test to get back to a green state
  5. Refactor the code to add what’s missing (and use other tests if needed)
  6. Uncomment the test
  7. Repeat from step 2

When doing this at the scale of a real life story or feature, we’d use git stash instead of comments. This way of working has a name, it’s called the Mikado Method. It is at the heart of making baby steps work in real life.

💡 The Mikado Method is at the heart of making baby steps work in real life

Take a break

With TDD and the Mikado Method we can put the refactoring on pause. We can perform a small increment of the refactoring, commit and deploy it … and pause ! We’ll work on business features for a while, and resume the refactoring later on.

A cup of coffee next to a computer. Developers can pause their large scale refactoring if they work in small enough baby steps

When done well, it feels slow. We have to remember that the alternative is to convince business people of prioritizing a refactoring … As we’ll regularly ship baby steps of a large scale refactoring, we’ll know we’re on the good track !

More to come

Unfortunately, even with bandwidth and skills, we are not there yet … It’s one thing for developers to do incremental software development of large scale refactoring on their own. It’s another to do it as a team !

This was the seventh post in a series about how to get sponsorship for large scale refactoring. Next post will be about how to manage constant merciless refactoring and baby steps as a team.

Incremental Software Development Strategies for Large Scale Refactoring #1 : Constant Merciless Refactoring

My previous post advocated incremental software development for large scale refactorings. It’s less risky and it prevents tunnel effects. Most of all, it’s a lot easier to convince business people of engaging in a refactoring this way.

It’s one thing to understand why it’s the way to go, but it’s another to be able to do it ! In this post, I’ll start by explaining how to find the time to do constant merciless refactoring.

This is the sixth post in a series about how to get sponsorship for large scale refactoring. If you haven’t, I encourage you to start from the beginning.

Constant Merciless Refactoring illustrated as a recurring cleanup activity

Steal Take the time for constant merciless refactoring

If it hurts, do it more often ! Wisdom of the internet

As a child, I used to be a very untidy kid. Every few week, my room would get in a real mess, and my mum would order me to clean all this mess. I would then lose 1 or 2 hours tidying my room up. She used to tell me that if I kept things tidy as I used them, I would not have to lose this time. From the top of my 10 years old, I would judge this advice as nonsense. 

This is the kind of mess my bedroom used to be in, when I was a kid, before I learned the virtues of constant merciless refactoring

Fast forward a few years, I am myself a parent and I’ve been working with legacy code for many years. These years have taught me how much I was wrong …

💡 The easiest refactorings to negotiate are the ones we don’t have to talk about !

The more refactoring we embed in ‘Business As Usual’, the more we’ll do, and the less we’ll argue with the business. We can wonder if this is still ‘professional’ ? In The Art Of Agile Development, James Shore explains that managing technical debt is the key to long term productivity. (Details in the Risk Management section of his book). As developers, we are the only experts in the code, the responsibility to keep it clean falls on us.

Never ask the permission to do a good job ! Doc Norton

There’s more to constant merciless refactoring ! It also keeps us in a good position to accept new features or refactorings later down the road.

Following are the 3 practices that make up constant merciless refactoring.

Yesterday’s weather and slack time

20 years ago, a promises of agile software development was to stick to a sustainable pace. When we are working with a flavor of Scrum, we can rely on it’s literature to inject some slack time.  Slack time is buffer time at the end of every iteration. We should not plan any work during the slack, to accommodate with the unexpected. It’s a way to deliver on forecasts, whatever happens.

In short, if your velocity chart looks something like that :

Drawing of a fluctuating team velocity. This is often the result of not enough constant merciless refactoring

Scrum tells us to plan what you delivered in your worst iteration for the next one ! When things will work bad, we’re still pretty likely to deliver what we wanted. When things work well, we’ll have time to tackle refactoring.

There’s a lot more to say about slack time. How to take some when you are using Kanban ? How to make sure you keep some when your velocity becomes pretty stable ? How to do you increase your velocity in the long term ? (I guess I’ll have to write a full blog post about this some day.)

The Boy Scout Rule

I already blogged about the Boy Scout Rule. Here is how Uncle Bob wrote it :

Always leave the file you are editing a little better than you found it. Bob Martin

Following this simple rule goes a long way to keep the code clean and ready for larger refactorings. It works arm in arm with Yesterday’s weather principle. The extra time we take for clean up impacts our capacity to plan stories and features. This creates time to keep on doing the boy scout rule in future iterations.

How ‘clean’ the code should be is a team decision. Coding conventions and a static code analyzer are very important to track the boy scout rule. I learned that code reviews, pairing, mobbing and coding dojos are great to agree on coding conventions.

Embedding refactoring in features

The Test Driven Development loop goes as Red-Green-Refactor.

TDD's Red-Green-Refactor loop. Itself highlighting Constant Merciless Refactoring as a recurring activity

The same loop goes on at larger scale for Acceptance or Feature Test. When repeated many times, the loop could as well be Refactor – Red – Green. In fact, it’s a lot easier to refactor when you know what feature you want to build than at the end of the previous one. (Nat Pryce wrote about that in more details)

💡 “Disguise” refactoring as first technical sub tasks of features to get them prioritized.

All this to say, we should start our features with the pre-requisite refactoring ! We should not build features on shaky foundations. We should also impact our estimates. In fact, it’s a lot easier to justify to business people at that moment. We don’t need to mention ‘refactoring’ or ‘clean up’. We can use technical sub-tasks to track these technical refactorings. Technical sub-tasks are the team’s and don’t need to be understandable by business people.

Technical tasks drawn "Under the sea" and visible to devs only. This leaves them room to do constant merciless refactoring

To be continued

Finding the time for constant merciless refactoring is one thing, but how do we fit the work in these short slots ? In the next post, I’ll continue about how to actually work in baby steps.

This was the sixth post about how to get sponsorship for large scale refactoring.

Incremental Software Development for Large Scale Refactorings

My previous post was about the badass developer attitude. More specifically, how it can buy sponsorship for large scale refactorings. Unfortunately, attitude is not enough. We also need to be able to deliver in a way that builds trust with the business. Most of all, business is scared of the tunnel effect. Incremental software development techniques allows to deliver large scale refactoring step by step. Not only that, but it also allows to do so alongside business features. That’s how badass developers walk their talk and gain the business people’s trust.

This is the fifth post in a series about how to get sponsorship for large scale refactoring. If you haven’t, I encourage you to start from the beginning.

Drawing of a plant at different stages of growth, illustrating Incremental Software Development

Why does incremental software development matter ?

A short story

A few years ago, I joined a team whose work involved a Domain Specific Language and a parser. The parser had grown in an ad-hoc way for a few years, and was now both very brittle and difficult to extend. We knew the way to go was to adopt a more solid parsing approach. We wanted to migrate to ANTLR and a multi pass parser.

As always, the business was very pushy for new features. There was no way we could have stoped everything for a full parser re-write. We explained to them that some of their features would be impossible to write without the new parser. We suggested that we work on migrating the parser as a background technical Epic. We did so using incremental software development techniques.

It took us a few months to migrate the parser. Meanwhile, we kept the software in a releasable state. This allowed us to validate our progress on the refactoring. We could continue to release features. We were able to share our progress on the refactoring with  the business people. They were very happy with the way we did this refactoring. In fact, it set a standard about how to prepare the software for big features later on.

The real problems

To understand why incremental software development works, let’s understand the alternatives’ problems. The main alternative it to do the refactoring in one massive task. This kind of initiative screams “Tunnel effect waiting to happen” ! The tunnel effect scares business people for 3 reasons :

  1. Because they don’t know how much money they’ll need to put in the tunnel to get out of it
  2. Because they don’t know when they’ll get the other features which they are also waiting for
  3. To be blocked in the tunnel if something unexpected and important comes along

Picture of someone holding a light, alone in a tunnel. Incremental Software Development helps to avoid the tunnel effect when performing a large scale refactoring

Delivering a large scale refactoring with incremental software development fixes these 3 points.

  • Every commit is a step forward while keeping the system in a releasable state. If ever something unexpected comes along, we can pause the refactoring for a while. (point 3)
  • Not all the team has to work on refactoring at the same time. Working on the refactoring does not block the delivery of other features. (point 2)
  • Finally, after working on a refactoring for a while, it becomes easier to have an idea of how long it will take. (point 1)

💡 Incremental software development fixes the business people’s fear of refactoring tunnel.

It is true that performing the refactoring in one team-wide batch would be more efficient. It would reduce the overall Work In Progress and get it done quicker. Unfortunately, it’s also a lot more scary for business people !

Incremental Software Development techniques

Like any skills, we can learn these techniques. Some are easy, and we can learn them from books. Others are more difficult and are usually learned through hard won experience. They can also be learned from a veteran developer in your team who’s been through all this already. If you aren’t or don’t have a veteran around, we can still manage. Deliberate practice is a great way to learn almost anything. Coding dojos are the thing here (I’ll write more about this later).

Once we master these skills, a lot of things change. First, we can do refactoring without harming our relationship with business people. Second, it builds enough self confidence among developers to negotiate with business people. This in itself, makes us more credible in their eyes. As a result, they are more likely to compromise on prioritizing refactoring.

💡 Mastering incremental software development builds self-confidence for developers.

To be continued

This was the fifth post about how to get sponsorship for large scale refactoring. In the next posts, I’ll deal headlong with the actual techniques. How to get bandwidth ? How to work in baby steps ? How to track the progress ? How to deal with the large scale ? Finally how to go further ? As you can see, there is still a lot to come, so stay tuned !

Principles That Will Make You Become a Badass Developer

In my last post, I went over things we should avoid if we want to become badass developers. As I said though, this is far from enough. Once we’ve stoped losing trust from the business, it’s time to build some more ! This is the forth post in a series about how to get sponsorship for a large scale refactoring. If you haven’t, start reading from the beginning.

The badass developers gain the business’s trust by sneaking in as business partners. A good way to become one is to start acting like one ! Here are examples of principles for that.

Arm of a badass developer with the tatoo '> Badass Principles;'

Keeping the business best interests in mind

This is all about decision making. We should try to steer decisions according to the business. Whenever we talk with business people, we should stay aways from technical topics. I still remember my younger self explaining threading details to a trader … Most of all I remember the look on his face ! We should avoid technical bla bla, but we should be clear about the business consequences.

Honesty and Candor

When we don’t agree with something, we should say so. When we don’t understand something, we should ask the question. We need to stick to facts and assume everyone has the business’s best interests in mind. Candor is a way to get our opinions and questions through, without sounding rude or pushy. There’s a whole chapter about candor in Creativity.inc, the book about Pixar.

Cover of the Creativity.inc book. It contains lessons on Candor we should all read to become badass developers

With time, business people will think of us as a positive and pragmatic problem solvers. That is exactly the kind of people they want to work with !

💡 Candor is a way to get our opinions and questions through, without sounding rude or pushy.

Strong opinions, but weekly held

Jeff Atwood, already wrote about this. The idea is to fight for our opinions, but let them go without a fuss when we proved wrong. We know that we are all very often wrong. Only fools or self-centered people don’t admit this reality. Business people won’t trust us in either case. We need to show that we can go over our previous opinions. This grows our reputation of rational problem solver.

Acknowledging when we don’t know

The same logic goes with knowledge. None of us knows everything. We have to admit when we don’t know something and ask for help. This proves that we place the business’s speed over our personal ‘know-it-all’ reputation.

Here is a story that happened to me at my first job. I’m sure most developers go through it one day or another. I was assigned a new task involving technologies I did not know. I did not have the guts to state upfront that I would have to learn them. The result was that I sent 2 full weeks fiddling with this task to get something working. The more it went on, the more the product people were wondering why it was taking so long, and the more I got stressed !

Be bold and say No !

If we are sure something we should not do something, we need to say so. Badass developers are not afraid to say they won’t do it. Good software engineering requires merciless prioritization. If there are doubt about the value of doing something, it’s often a better idea to make sure before wasting time on it.

There are many ways to say ‘No’. Before giving a harsh ‘No’, we can try to challenge decisions. We can ask for clarifications and rationals through open questions. Very often, highlighting the risks makes people review their plans. As technical experts, we should also share as much of the consequences as possible.

In the end, badass developers are ready to leave a FUBARed situation. Great engineers don’t have troubles finding jobs these days … There’s no reason they should accept to be waisting their time.

💡 In the end, badass developers are ready to leave a FUBARed situation

What do to next ?

As we become badass developers, our reputation will grow. We’ll be in a better position to negotiate a large scale refactoring with the business. There’s a catch though : we’ll need to live up to our reputation ! Admitting that we are wrong 100% with candor will not make it ! 

When we manage to negotiate a large scale refactoring, the team will need to do a good job of it. This boils down to delivering it piece by piece, alongside features. This is exactly what my next post will be about.

This post was the forth post in a series about how to get sponsorship for a large scale refactoring.

5 Mistakes Badass Developers Never Do

Here is a one sentence summary of my previous post.

Badass developers negotiate large scale refactorings with the business better.

Unfortunately, not all of us are sitting next to a true badass developer … Hopefully, we can all become one ! Depending on our track record, it’s going to be more or less difficult, but with time and the good attitude, we can all do it. Becoming a badass developer is all about trustworthiness.

This post is the third in a series about how to get sponsorship for a large scale refactoring. If you haven’t, I recommend you to start from the beginning.

The first thing to become trustworthy is to avoid things that kill trust. Sounds obvious, but it’s very easy to forget. Here are 5 examples of trust killers you should never do if you want to become a badass developer.

Drawing of a hurt finger after someone made a mistake with a hammer. Badass developer don't do this kind of mistakes !

Resume Driven Development

We should pick the best tools for the job. The best tools are often a bit old, precisely because they’ve been battle tested in production for a while ! That’s exactly the kind of technologies you want your business to rely on.

To keep his skills up to date, a badass developer will not add a funky new tech in the production code. He would rather negotiate slack time with the business. He might also openly take his Friday afternoons to experiment the latest techs ! He would simply explain that it’s to avoid polluting the production system.

💡 A badass developer will not add a funky new tech in the production code.

Over-engineering

Gold plating or over-engineering are similar anti-patterns. A badass developer always keeps the business’s interest in mind. This means he knows how to balance long term design and short term features. A good rule of thumb is to keep a vision in sight, but to get there in baby steps.

Build features with no agreed upon value

Product managers are here to select what should and what should not be in the product. As product experts, they are the ones who know how much a feature is worth. Except (maybe) when we are building tools for others developers, they know better than us. Adding something of dubious value in the product is bad in two ways. 

  • First, it takes some time to build, time that we could use to build valuable features instead. Remember : Real developers ship !
  • Second, it creates unnecessary code to maintain. In the worst case, it can constraint the architecture. Which might eventually prevent us from adding other more valuable features afterwards.

Hide in a tunnel

We should always be careful of the tunnel effect. Seeing their money vanishing with no visible output makes business people, understandably, creepy. As soon as things become too complicated, a badass developer will raise the alarm. The fact is that he has been in this kind of situation before, and knows how to recognize it. At that point, it’s still possible to discuss the problem with everyone, and adjust the plan.

As an interesting side note, I was at the Paris DDD Meetup last Thursday. We had the chance to welcome Eric Evans as a surprise guest ! When asked what were his worst mistakes, he said something along this line :

💡 Some of my biggest mistakes were not backtracking soon enough a few times as I was drifting in quagmire. Eric Evans

Eric Evans, the father of DDD, a true badass developer, answering questions at the Paris DDD meetup


Let the team down

It’s not only about getting the business’s trust. We must also build trust from our fellow developers. Whenever we break the build and leave, or worse, deploy and leave, that trust is gone for a long while… We should not do that !

There’s more to a badass developer

I’m done with this list of things badass developers don’t do. Avoiding these is only the first step to become a badass developer. In next post, I’ll write about what we need to do if we want to build strong trust with the business.

This post is the third post in a series about how to get sponsorship for a large scale refactoring.

Why We Need Badass Developers to Perform Large Scale Refactorings

My last post was about the challenge for dev teams to get sponsorship for large scale refactorings. I listed two main reasons :

  1. The business doubts developers to have their interests in mind
  2. They are also not aware of the cost of the current technical debt

This post (and the next) will be about how to gain the business’s trust. This is exactly where badass developers can help. Let me start with a story.

Drawing of 2 hands of a badass developer over his keyboard, with ">badass<" tatooed on his fingers

Back in 2002, at my first job, I started to read Refactoring, Improving the Design of Existing code. That’s where I read about unit testing. I found it so great that I made a demo to other programmers. Being the junior dev in the team, my co-workers reaction was something between “Meh” and “Maybe”. Fortunately, a more experienced and respected developer gave it a try. A few weeks after my demo, he announced to the team that unit testing worked great on new code. This time, people showed no questioning about his opinion : he if said so, it must have been true. Even business people blessed the practice !

The "Refactoring, improving the design of existing code" cover. Badass developers know how to perform large scale refactoring

I had given a good live coding demo, but it was this respected developer’s opinion that won the point. To convince business people of sponsoring a large scale refactoring, we need their trust. That’s why we need badass developers around.

💡 I had given a good live coding demo, but it was this respected developer’s opinion that won the point.

What is a badass developer

Badass Developer's fist with a ring "I am badass"

By Brooke LarkCC0, via Wikimedia Commons

Badass developers are first of all people who are credible to the business. This usually implies a track record of delivering features and refactorings. Badass developers understand the business constraints. That’s why they learned how to deliver refactorings alongside features. They also need to be responsible and bold enough to stand ground in front of the business. Finally, badass developers are able to train others.

💡 Badass developers are first of all people who are credible to the business

Unfortunately, there are not so many badass developers in the industry … It has a youngster bias, and tends to push experienced developers to other activities. As if 10 years of systems design was less valuable than knowing the latest language !

Learn more about Badasss developers

I tried to find other words before resorting to ‘Badass’. Unfortunately, I could find none that got the point so clearly. Uncle bob calls them ‘software professionals’ in The Clean Coder. Professionalism is not specific enough to me. Adam Nowak also calls them ‘responsible developers’ in a blog post.  That does not convey the idea that, sometimes devs need to stand guard in front of the business.

The clean coder book cover. Clean coder looks like a form of badass developer

These concepts, though, are very close to my definition of a badass developer. Check by yourself :

To be continued

This was why badass developers matter to the success of large scale refactorings. This was the second post in a series about how to get sponsorship for a large scale refactoring.  In the next post, we’ll look at what we can do to all become Badass developers.

How to Convince Your Business of Sponsoring a Large Scale Refactoring

Whenever I present or suggest a good practice to dev teams, I often get the same remark. Here is how it goes :

  • That’s a great idea and we would love to do this, but our code is in such a mess that we cannot !

  • Maybe you should start doing more refactoring then !

  • We would like to, but we don’t have the time. We are fire fighting all the time.

It’s a bit like the old adage of the lumberjack that is too busy to cut wood to sharpen his axe… The sad part here, is that most of the time, developers know they would be a lot faster if they could clean up their code. Unfortunately, they are usually not given the time.

How do we end up in this silly situation ?

Drawing of a '5 whys' mind map explaining why it is difficult to get sponsorship for a large scale refactoring

Only developers see the bad code

As I’ve already been joking about, code is invisible. Mess in the code even more so, especially to people who don’t code. The code could look like that and no one would notice.

Inside of a kitchen from someone suffering from Diogenes syndrome

Par Un TouristePhotographie personnelle, CC BY-SA 3.0, Lien

If someone put his own office in that state, he would get fired, but not for the source code. The good side is that we, developers, are safe, we can continue to wreak chaos without fear ! That’s pretty weird when we think that this is what we ship to customers …

💡 Is Diogenes syndrome for source code a recognized pathology ?

Business might also not see bad code because that’s the only thing they’re used to ! Maybe they’ve always been working in dysfunctional organizations that systematically create crappy code. Slow teams, late deliveries and fire fighting might be business as usual for them. From this point of view, trying to improve code is a pure waste of time and energy. The same goes for large scale refactorings.

The worse part of all this is that if devs don’t have the time to keep their code clean, it will only get worse. This will reinforce the view that software delivery is slow and that there is nothing to do about it !

Business has been burnt in the past !

Bad experiences are another reason why business is unwilling to sponsor refactoring. Did someone sell them an unrealistic productivity boost that turned in a never-ending tunnel project ? Badly managed large scale refactorings deliver late, create no value, and a lot of bugs. At one company I worked for, business gave devs 1 full year (!) to clean up the code … We took 2 !! Meanwhile, the CEO had to dilute the stocks a bit to keep the boat afloat ! I’d think twice before giving this kind of mandate myself.

Performing a large scale refactoring is not easy, and involves specific skills. These skills are about refactoring in baby steps, alongside feature delivery.

Usually, people acquire these skills through hard won experience … Unfortunately for us, our industry is not very nice to experienced engineers … It’s a lot easier to hire a fresh grad who knows the latest javascript framework than a 2 decades engineer. (Who, BTW, could learn this framework in 2 weeks …) It’s also a lot harder for the junior developer to succeed in negotiating a refactoring.

Again the twist of fate is that junior engineers are a lot more likely to start a submarine latest-framework.js rewrite supposed to solve all maintenance issues … which will only make things worse.

Overestimate, only as last resort

A quick fix is to systematically overestimate to get enough time to refactor. As any other ‘submarine’ initiative, I would recommend it only in last resort, after you’ve tried every other possible technique … and just before you quit.

Hiding things to the business people kills trust and hides problems. Trust and collaboration is what you need to get the business to sponsor large scale refactorings ! Plus, if ever you mess up (as submarine initiative often do) you’ll be the only one to blame …

That said, ‘overestimating’ so that you can write clean code is ok. It’s not overestimating, it’s estimating to do a good job.

💡 We should never ask the permission to do a good job. (Doc Norton)

To be continued

You might wonder what these other techniques are ! That’s exactly what I’ll go through with the next posts. This was the first one in a series about how to get sponsorship for a large scale refactoring. The series will cover topics like :

  1. How to convince your business of sponsoring a large scale refactoring
  2. Why we need Badass developers to perform large scale refactorings
  3. 5 mistakes badass developers never do
  4. Principles That Will Make You Become a Badass Developer
  5. Incremental Software Development for Large Scale Refactoring
  6. Incremental Software Development Strategies for Large Scale Refactoring #1 : Constant Merciless Refactoring
  7. Incremental Software Development Strategies for Large Scale Refactoring #2 : Baby Steps
  8. Incremental Software Development Strategies for Large Scale Refactoring #3 : Manage it !
  9. Incremental Software Development Strategies for Large Scale Refactoring #4 : Patterns
  10. Nothing convinces business people like money

How to Avoid Unnecessary Meetings (a Takeaway From Devoxx France 2018)

I had the chance to attend Devoxx France this year in Paris. Here is the most important lesson I learned :

How to avoid unnecessary meetings with asynchronous decision making

Bertrand Delacretaz, a member of the Apache foundation. He gave a great talk about how the open source community handles decision taking. Open source developers are often all over the world, often in different timezones. Meetings are not an option for them. Still, they manage to make great decisions !

Drawing of a decision hammer

Even if you don’t work remotely, avoiding unnecessary meetings is always a great thing !

  1. You’ll have more time to do productive and interesting stuff
  2. You’ll avoid interruptions and be even more productive
  3. If you are an introvert, it’s going to be easier to contribute to the decision
  4. As people have more time to think through the decision, the result is usually better

For a full walkthrough, I encourage you to watch the talk in full length. If you don’t speak french, an english version is available here. Finally, slides are also available in french and english.

💡 Even if you don’t work remotely, avoiding unnecessary meetings is always a great thing !

Crash course

For the hasty folks among you, here is a summary. The decision making follows 4 stages :

  1. Open discussion and brainstorming. People discuss openly and suggest ideas in a free form manner.
  2. Emergence of options. After enough discussion, a few options will start to make more sense than others.
  3. Coming to a consensus. Someone will draft a formal proposal. People will discuss and amend this proposal until they reach consensus. Consensus is not unanimity !
  4. Decision. After consensus, the benevolent decision owner validates the decision once and for all.

Until the decision is taken, the process can move forward but also backward.

Tooling

We need only two tools to make this possible :

  1. For discussion, brainstorming and emergence of options, use a very open and chatty tool. The speaker called this a “shared asynchronous communication channel”. This can be an online chat, a mailing list or Github issues (ex). It could even be a real life whiteboard if you all had access to it.
  2. From drafting the proposal to the end, prefer a structured and chronological tool. The speaker suggests using a “shared case management tool”. Draft the proposal in this tool, and use comments to log the latest steps of the decision taking. He had examples using Jira issues (ex) or Github pull requests (ex). To confirm the decision, close the case. The tool will record which version of the decision was exactly taken.

Architecture Decision Record

Drawing of an Architecture Decision Record which work great with asynchronous decision making

ADR is the practice of documenting architecture decisions. It makes sure we remember why we took a decision. This can be very useful to know how to deal with the existing software. A widespread practice for ADRs is to use simple text files in git. There are even tools for that. This looks like a perfect fit for decision making using git pull requests ! I’ll write a post about that when I get the chance to try.

💡 Git pull requests based asynchronous decision making is a perfect fit for Architecture Decision Records.

Currently experimenting

I am currently trying this whole decision making technique at work. We are still in the brainstorming phase. We are using our internal chat app for that. Options are starting to emerge, but we did not move to the consensus part yet. I’ll write a return on experience post when we reach the end.

A Coding Dojo Exercises Plan Towards Refactoring Legacy Code

My current job at work is technical coach. I’m available for teams that need help to adopt incremental coding practices.

Problems with refactoring legacy code

A few months ago, a team which was struggling with a lot of legacy code asked for help. As you might know if you read my blog, I’m a big fan of Test Driven Development (TDD) because it has made my life as a developer so much more easy. I’m so used to TDD now, that even if I don’t have tests yet (as is the case when refactoring legacy code), TDD helps me :

  • To stick to baby steps which are a lot less likely to fail than larges changes.
  • Write testable code. I know what testable code looks like, and when refactoring, I’ll try to change it towards that.

That’s why we started to run regular, all team, coding dojo randoris. It was nice for the team dynamics, and the people where learning a lot of technical skills. I also got the feedback that they where not able to apply this directly on their day to day job though. After a bit more discussion, I understood that they did not know where this was going, what to expect, and when !

💡 Test Driven Development also teaches you what testable code looks like.

The coding dojo exercices

It turned out that a coding dojo exercises plan was enough to answer their questions. This is what it looks like.

Drawing

An illustrated Coding Dojo Exercises plan leading to the mastery of Legacy Code Refactoring

Mind Map

Here is another, more concrete, version, with sample names of katas we can find online.

An mind map of Coding Dojo Exercises plan leading to the mastery of Legacy Code Refactoring

Text

It starts with simple greenfield katas :

It goes on to intermediate katas, where we can use TDD to do design :

From then on, it’s possible to tackle advanced katas and styles :

All this opens the gate to legacy code refactoring katas :

At that point, the team can mob to refactor production code :

  • Real life, static analysis issue, mob programming session
  • Real life, code smell, mob programming session
  • Real life, larger mob Refactoring

What changed in practice ?

We wanted to split the teamwork and the coding dojos exercises. The team is now doing mob programming sessions on their usual stories twice a week (I’ll blog about that someday). But also doing regular coding dojos exercises in pairs.

Even if they did not go through all the TDD katas yet, mobbing on real stories helps the team to take on legacy code.

Given enough eyeballs, all bugs are shallow. Linus’s Law

Working in pairs on the code katas allows them to be more engaged in the exercises. In the end, it brings faster learning.

💡 A mix of Coding Dojos in pairs and Mob Programming sessions is a good way to teach TDD in a Legacy Code context.

When Is Testing Using Mocks Still a Good Idea ?

In the previous 7 articles of this series, I’ve tried my best get rid of mocks. I’m pretty sure that using these techniques will get you a long way out of mock hell. Excessive mocking leads to unmaintainable tests. Unmaintainable tests lead to low coverage. Low coverage ultimately leads to legacy code. If you haven’t already, I encourage you to start reading from the beginning.

One question remains though : Is it realistic to get rid of all mocks ? An even better question would be : Are mocks always bad ? Are there situations when mocking is the best choice ?

When mocking still makes sense

Let’s to through a few examples.

Testing a generic wrapper

A few years ago, I had to write a service for an enterprise system. As any service, I had to ensure that it was returning nice errors. We decided to capture and wrap all errors from a few ‘gate’ points in the code. We built a generic wrapper that did only delegation plus exception wrapping. In this case, it made a lot more sense to test this with a mocking framework.

1
2
3
4
5
6
7
8
9
10
11
context ServiceErrorWrapper do

 specify 'converts all kinds of exceptions' do
   failing_object = object_double("Failing object")
   allow(failing_object).to receive(:long_computation).and_raise(Exception.new("Something terrible happened"))

   expect{ ServiceErrorWrapper.new(failing_object).long_computation }.to raise_error(ServiceError).with_message("Something terrible happened")
 end

 # ...
end

Not only did we reuse the wrapper many times in my service. We also ended up using it in other services as well !

Injecting a hand written in-memory fake

As you might have noticed, in the previous article, I recommended to use an in-memory fake instead of mocks. By nature, an in-memory fake is a kind of mock. Even if it is not defined by a mocking framework. (I actually think that by making mocking so easy, mocking frameworks often do more harm than good.)

💡 By making mocking so easy, mocking frameworks often do more harm than good.

Still, I used const_stub(...) to inject the in-memory fake.

1
2
3
4
5
config.before(:each) do  

  stub_const("TwitterClient::Client", FakeTwitterClient.new)  

end  

I did this for 2 reasons :

  • Production code can continue to use a straightforward constant
  • I don’t risk forgetting to remove the mock at the end of its lifecycle, the framework does this for me
  • As I’m injecting the same fake for all tests, there is not much risk of test conflict (for the moment)

Testing a cache

The “raison d’être” of a cache is to avoid doing something twice. It should also return the same results as if it was not there. This is by nature almost impossible to test with state based assertions. Mock frameworks are great for this situation though. Here is an example :

1
2
3
4
5
6
7
8
context "UsersController" do
 it 'caches users' do
   expect(User).to receive(:load).once.and_return(User.new(name: "Joe"))

   controller.login('Joe', 'secret')
   controller.login('Joe', 'secret')
 end
end

The assertion could not be more explicit, we are checking that the expensive load was only done once.

Legacy code

Michael C.Feathers explains that testing using mocks is a key practice in "Working Effectively with Legacy Code"

In Working Effectively with Legacy Code Michael Feathers explains how to exploit “seams” in the code to put it under test. Mocking is straightforward way to inject behavior through a seam.

Mocking is a pretty good starting point but we need to be careful and keep a few things in mind. Legacy or not, we must not forget that too many mocks will make tests unmaintainable !

  • It’s a good idea to refer to a target design or architecture blueprint to know where to inject mocks. (I’ll write a post about this one day). This increases the chances to replace them with an in-memory fake later down the road.
  • Plan to replace the mocks with a better design as soon as possible.

It depends …

As with anything in software, there is no absolute rule about mocking. Even if I prefer not to 99% of the time, there are situation when testing using mocks is the thing to do. Knowing the risks, it’s up to you to decide !

If using a mock, prefer spy / proxies

Spies and proxies make testing using mocks less intrusive

As I explained in previous posts, mocks duplicate behavior. If we could use mocks without duplicating behavior, they would do less harm.

It turns out there is a flavor of mocks for that : spies and overlooked proxies. Proxies do the real thing but also record the calls and return values. It’s as non-intrusive as mocks can be.

💡 Proxy mocks are as unintrusive as mocks can be.

For example, here is how our cache test would look like using a proxy :

1
2
3
4
5
6
7
8
9
10
context "UsersController" do
 it 'caches users' do
   allow(User).to receive(:load).and_call_original

   controller.login('Joe', 'secret')
   controller.login('Joe', 'secret')

   expect(User).to have_received(:load).once
 end
end

It’s more verbose, but simpler. Most mock frameworks provide some form of spy or proxies. A few years ago, I also wrote rspecproxies, a wrapper on top of rspec to make this easier.

This is the end

This was the 8th and last post in a series about how to avoid mocks. Before closing here is a list of other references about the topic.