5 Effective Warning Signals That Will Get You Sponsorship for a Large Scale Refactoring

In 2005, professors Bizer and Petty showed something interesting about human behavior. People make more efforts to avoid what they don’t want, than to get what they would like. The study itself is interesting, you can have a look at it here. For example, it explains why political campaigns are getting more and more nasty. There’s also a lesson for us, mere developers. We’ll get more sponsorship for our refactorings if we highlight the dangers of not doing them !

Drawing of a warning panel with legacy code written below an awful monster. Sending effective warning signals is a good way to get sponsorship for large scale refactorings

This is the 11th post of a series about how to get sponsorship for large scale refactoring. If you haven’t already, start by the beginning.

From the inside, many systems are in such a messy state that seem like a catastrophe waiting to happen. Unfortunately, this mess is completely invisible to non-developers ! Here are a few techniques to show how close we are from a total breakdown !

Effective Warning Signal #1 Pranks

If you have the guts and your company is fun enough, you can try one of these pranks 😉.

A photo of a desk littered under paper. As if we had printed the whole source code and dumped in on a non-developer desk. Pranks are a good way to get sponsorship for large scale refactorings

What if we printed the whole source code and dumped in on non-developers desks ? By Kris Krüg, CC BY-SA 2.0, Link

They’re bound to have a big impact … but they might also get you fired ! We should be creative and find both effective and acceptable pranks. Pranks are a lot more effective than we first think. Non-Violent Revolution activists have actually used Laughitism to take dictators down ! For a good (and unexpectedly fun) read on the topic, have a look a Blueprint for a Revolution. It was written by Serb non violent activist Srdja Popovic) member of OTPOR!, who brought Millosevic down .

Cover of the book Blueprint for Revolution. Laughitism is a non violent technique that we can use against Legacy Code to get sponsorship for large scale refactorings

Promised, as soon as I manage to use such a prank without getting fired at work, I’ll blog about it !

Effective Warning Signal #2 Dice of Debt Game

While doing my researches for this article, I found the Dice of Debt game. It’s aimed at making business people experience the long term legacy code drag. It has good reviews, have a try at it with your business people and post back your feedback ! I’ll do so as soon as I have a chance to test it myself.

Effective Warning Signal #3 Higher authority

Appealing to a higher authority works as long as people recognize this authority as so. Knowing the people in front of us, it’s our job to bring up the reference in an effective way. Here are

In this talk, Doc Norton, a recognized technical debt expert, shows the link between technical debt and productivity.

Slide from Doc Norton's Technical Debt Trap talk. It shows how technical debt impacts the cost of change and can be an effective warning signal for a large scale refactoring

Showing similarities between our own code metrics and this graph might ring the alarm bell in business people.

Another interesting model out there is A2DAM. It was built through the Agile Alliance. It can be used to estimate the value of a codebase when buying a company. Maybe business people will listen if we tell them that their software would be worth 0 on the market ?

Effective Warning Signal #4 Metaphor

I was recently working with a team that is preparing a pitch to get sponsorship for a refactoring. They want to rework multithreaded code that uses locks and other low level synchronization. The hand written synchronization is becoming difficult to maintain. They would like to refactor it with the actor model. Business people will likely argue that this module is now stable enough and should stay as is. We all know that this is not the case with bogus multithreaded code : the more you use it, the more bugs you find ! They had the idea to use the email vs phone metaphor. No one in the room could have handled all his daily emails with a phone only ! Everyone understood why it was necessary to switch to actors.

In A Taxonomy of Technical Debt, Bill Clark adds another dimension to technical debt. On top of the classic principal and interests he adds contagion. In fact, it’s a bit as if he’s ditching the technical debt metaphor for the disease metaphor ! Disease have a cost to live with, a cost to heal from and a contagion rate. People at the agile alliance also noted this self reinforcing behavior. This metaphor might be better for “cruft”. Ward Cunningham’s original metaphor of technical debt only applied to tested code.

💡 A disease might be a better metaphor than debt for code cruft.

Effective Warning Signal #5 A horror story

We said that a successful refactoring story will be useful to frame ours as an opportunity. We can have more impact with the opposite ! We should relate a large software failure, that had impact on the business to bad code. The bigger the impact on the business, the better it is. If you (unluckily) have something like that at your company, it should be a very powerful argument. If you don’t, try to find a public story on the internet, or a public conference. For example here is one from The 10 Worst Programming Mistakes in History. 

💡 The Therac-25 (a radiation therapy machine) killed 6 people because it was difficult to perform automated tests !

We can draw parallels and forecasts, to highlight the high risk of failure.

Almost there

This was the 11th post in a series about how to get sponsorship for large scale refactoring. Unfortunately, presenting refactoring in a good way only brings us so far. If we want to be really convincing, we need to use quantitative data. That’s going to be the topic of my next post.

Presenting a Large Scale Refactoring as a Business Opportunity

Have you ever tried to talk about refactoring with business people ? Most of the time, the matter is pushed aside or received with rolling eyes … A few weeks ago, someone on Hacker News asked the question “As a CTO, what is your most frustrating problem with technical debt?”. Here is the most voted answer

I think a lot of the time when a developer shouts “technical debt” what they are really shouting is “code someone else wrote that I’d rather rewrite than understand”. (The rest of the time is the same but they’ve understood it enough to think it’s a disaster area.)

I have found it’s best to not take tech debt complaints very seriously and instead look at actual success metrics. For example if every change to a bit of code introduces new bugs then that might be a reason to tidy it up.

ghiculescu

We need a constructive dialogue with business people to get sponsorship for important large scale refactorings. Let’s see what we can do to have one. This is the 10th post of a series about how to get sponsorship for large scale refactoring. If you haven’t start by the beginning.

Drawing of a tree growing on top of the word refactoring. The key to getting sponsorship for a large scale refactoring is to present it as a business opportunity

Presenting a business opportunity

We must be very careful about how we present refactorings. We don’t want them to be seen as unnecessary chores, or the latest tech fashion to follow. These don’t bring value, and business people will run away from such refactorings. Instead, we should present business opportunities for higher productivity to invest in.

Presenting a similar success story and its impact on the business

Most of our companies have been through similar refactorings in the past. We can try to find a successful one and draw parallels to forecast benefits for the business. If the company is too young to have any or they were all failures, we can have a look in the whole industry. When I was at Devoxx in Paris, Hervé Lourdin the CTO of VideDressing presented how they managed to do a large scale refactoring. Among other things, he went over how he managed to get sponsorship from his board. If you understand French, have a look at the full talk. In this case as in most, a likely promise of reduction in the costs of bugs and new features is what made the point.

💡 In most cases, a likely promise of reduction in the costs of bugs and new features is what gets a refactoring prioritized.

Be a Badass Developer

I wrote a lot about being a badass developer earlier in this series. This is when it becomes crucial. Being badass is a way to gain the trust of business people. Without this trust business people will react like the guy on Hacker News. Badass developers are way better at presenting large scale refactoring as business opportunities.

Find a path to do incremental delivery

Without incremental delivery, a refactoring risks delaying features for an unknown time. That scares the hell out of business people. Day to day incremental refactoring best practices will save the day here. They’ll actually help 3 times !

  1. To already perform a lot of refactoring in day to day work
  2. To learn how to find an incremental refactoring path to present to business people
  3. To prove that we know what we are talking about : we’ve already been doing it for a while

If you haven’t, have a look at the articles about incremental refactoring techniques that I wrote earlier in this series.

Pitch It !

In “corporate” environments, building credibility is a lot about being convincing. The more we learn to be convincing, the more likely we are to have our refactoring prioritized. If you are ready to spend some time learning how to pitch, I recommend reading Pitch Anything. At least have a look at its summary. It contains many actionnable nuggets to deliver powerful pitches. Here are a few.

Cover of the book Pitch Anything. Pitching a large scale refactoring as a business opportunity is a great way to get it prioritized

When we present an idea, we should put “frames” (time, authority …) in place to gain control of the discussion. For example, a time frame is a kind of deadline that will urge people to take action now. (Before overthinking it …) 

We should create tension by alternating phases where we are giving and phases where we take a step back. Following the same idea, it also mentions the Tao of Steve to rule at negotiations :

  • Don’t want anything
  • Show that you are really good
  • Leave at the crucible moment

Giving a great pitch is a great way to present large scale refactorings as business opportunities.

💡 Pitching is a great skill for developers to get sponsorship for a refactoring.

More to come

This was the 10th post in a series about how to get sponsorship for large scale refactoring. In next week’s post, I’ll go over how to use a recent discovery about how our brain works to become even more convincing !

Incremental Software Development Strategies for Large Scale Refactoring #4 : A Pattern Language

It can sometimes be a real challenge to integrate, let alone deploy, a refactoring step by step ! Here are some patterns that make this easier.

This is the ninth post in a series about how to get sponsorship for large scale refactoring. If you haven’t, I encourage you to start from the beginning. It’s also the fourth about incremental software development strategies for large scale refactoring. My point is that it’s not possible to sell a refactoring to business people until we master those :

  1. How to find the time to refactor in our daily work
  2. How to learn to work in baby steps
  3. How to organize and manage this ongoing effort as a team

What about when it is not easy to split the work into incremental steps ?

Do you remember the DSL parser refactoring story in mentioned in another post ? Switching to a different parser technology incrementally sounds like an impossible mission. Even so, that’s what we did ! 

Here’s another story. A long time ago, I was working in a bank. We were to migrate imperative C++ financial contract models into declarative definitions in C#). We had to maintain an ugly adaptation layer. It made it possible migrate and deliver step by step. In the end, we suffered almost no bugs resulting from this transition.

Drawing of the pattern for a T-shirt. Patterns can be useful for Incremental Software Development of Large Scale Refactoring

Why the effort ?

Incremental refactoring implies going through Frankensteinesk intermediate situations. Situations where both the legacy and the new models exist in the software at the same time. This kind of adaptation layer costs time and energy, but doesn’t add value to the product either ! What’s the point of going through this ? Isn’t a Big-Bang change cheaper ? Here is why it is still worth doing :

  1. It’s safer. With incremental delivery, we confirm that what we are doing is working in production. On top of that, if something goes wrong, as we only delivered a small increment, the problem is easier to diagnose.
  2. It’s also safer in term of priorities. The system keeps working during all the refactoring. There’s no pressure to finish it before we can move on to the next ‘valuable’ feature. As I explained before it makes it possible to pause, and why not stop there for the moment. This can be helpful if we hit a new urgent priority.
  3. Finally, it creates value earlier. Instead of having to wait 2 months to get all the value, you start getting a bit of this value every week. Even refactoring create value ! They reduce the time wasted to fix bugs. They increase our productivity. Sometimes, they even improve Non Functional Requirements of the system.

Plot of value over time in incremental vs big-bang. We can see that incremental software development creates more value.

As we can see, the goal of incremental refactoring is not only to work in small steps. We also want to find a way to deliver value incrementally !

💡 Incremental refactoring is not only about baby steps, it’s also about early value delivery !

A Mini pattern language

As I said earlier, this is easier said than done. Some piece of code won’t let you refactor them step by step easily. Some will be too obscure to know where to start. Some will be just huge. Others will depend on an all encompassing third party. etc.

Here is a short pattern language to deliver large scale refactorings incrementally.

Discuss with a domain experts

Goal

We need to refactor code containing a lot of domain knowledge

Conditions

We have a domain expert available

Therefore

Have regular discussions with the domain expert to find the best modeling possible.

Consequences

  • 💚 We get simpler code than by trying to replicate the twisted legacy logic
  • 💚 Can save a lot of work by skipping deprecated aspects.
  • 💚 Chance to get bug fixes or new features for free
  • ⚠️ The system does not exactly behave as it used to, which can cause integration problems

Difficulties

It can sometimes be difficult to find a domain expert …

A lot of the presentations at the July Paris DDD Meetup were about how to find domain experts. Who actually seem to be pretty rare beasts ! Here are my notes.

Mind map of my notes at July Paris DDD Meetup. The point was how to find a domain expert

Bubble context

Drawing of developer and business expert inflating a bubble together. The bubble context is a way to incrementally grow a large scale refactoring from within the software

Goal

We want to refactor a large piece of code with no Big-Bang

Conditions

We have access to the internals of the code to refactor

Therefore

  • Create a new bubble of clean code (a namespace, a package …)
  • Rewrite a piece of legacy code in the bubble
  • From the legacy code, delegate to the bubble
  • Repeat until the legacy code is not used anymore

Consequences

  • 💚 Enables a step by step continuous delivery of the new version
  • 💚 It is possible to eventually transform the API of the system
  • 💚 It’s easy to add new features in the bubble long before the refactoring is over

Difficulties

  • Need to understand the legacy enough to find good delegation points
  • Need to understand what the old small piece of code was doing to re-write it. A domain expert might be mandatory.

Strangler

The bubble context grows from the inside, but the strangler starts from the outside.

Goal

We want to refactor a large piece of code with no Big-Bang

Conditions

We can keep the same interface (API) for the legacy and the refactored versions

Therefore

  • Wrap the existing code
  • Re-implement calls in the wrapper
  • Delegate the rest to the legacy
  • Repeat until you support all the interface
  • Remove the legacy code

Consequences

  • 💚 Enables a step by step continuous delivery of the new version
  • ⚠️ Maintenance of the wrapper and both versions of the code during all the refactoring

Difficulties

  • Interaction between the legacy and the refactored version is not always as simple. For example when the wrapped code is stateful
  • The granularity of the steps is the (method) calls to the interface. They need to be small enough for the whole process to be incremental

Remember my story about how we switched our DSL parser to ANTLR ? We used a Strangler to do this.

Drawings of a strangler plant growing around legacy code. The strangler is very effective to do incremental software development of doing large scale refactorings

Feature toggles

Sometimes, we just don’t find a way to deliver a refactoring to users step by step. I’ve seen situations where all incremental paths implied a temporary impact on NFRs. Ex : Keeping both versions of the code implied an increase in resource consumption.

Drawings of an electric switch. Feature toggles are sometimes the last resort to do incremental software development of large scale refactoring

Goal

Incrementally build a refactoring that we cannot deliver piece by piece to all our users.

Conditions

When we cannot find a way to incrementally deliver our refactoring to bulk of our users

Therefore

From the code, dynamically switch to the different versions depending on runtime configuration. This way, we let most users stick to the legacy version. Yet, we can build, test, integrate and deploy the new version to beta testers.

Consequences

  • 💚 We can build, integrate and test our refactored code in baby steps
  • 💚 We can beta and A/B test our refactored code
  • ⚠️ We need to maintain and evolve both versions of the code for a long time
  • ⚠️ We need to maintain the switches in the code
  • ⚠️ We only deploy to beta testers, and don’t get as much early value
  • ⚠️ Duplicate the Continuous Delivery pipeline to test different feature toggle sets

Difficulties

Maintaining feature toggles is a mess. Thus, we need to

  • As much as possible, prepare the code to reduce the number of switches. Ref : Branch by abstraction
  • Hunt down the number of active feature toggles at any given time
  • Reduce the scope of toggles. Where possible, we should push things out of the toggle into stranglers or bubbles.

Feature toggles are an alternate to branches. Even if toggles are painful to use, branches are worse ! I’m not going to go over branches. If you want to see why we should not use branches, check this talk.

💡 Feature toggles are painful, but branches are worse !

Final word

I’m done writing about Incremental Software Development Strategies for Large Scale Refactoring. This is only what I currently know about this very important topic. There’s one last thing we need to do to be successful at it. We all need to keep an eye on new ideas from the community, and to share this with our teams as much as possible.

This was the ninth post in a series about how to get sponsorship for large scale refactoring. If you haven’t start by the beginning ! In next post, I’ll start to go over how to present a refactoring in financial terms to business people.

Incremental Software Development Strategies for Large Scale Refactoring #3 : Manage It !

My previous posts where about how to find and use small time slots for large scale refactorings. Refactoring step by step is a series of slack slots, sub tasks of features and boy scout rule increments. Unfortunately, keeping track of all these is a challenge of its own. Here are some organization best practices for that.

Here is a little story of what can go bad. I used to work in a team which had a high ‘refactoring culture’. Everyone in the team wanted to apply the kind of practices I mentioned in my previous posts. To make things more tricky, we were working from 2 cities. We had introduced slack time and developers would tackle refactoring at the end of every iteration. As we did not particularly organize or collaborate on slack time, we soon ran into conflicts. People wanted to refactor the same code, maybe in different ways ! After a while we also had too many large scale refactorings going on at the same time. This slowed down progress, increased the bus factor and the failure rate. Worst of all, it made it difficult to refocus on a newly discovered but urgent refactoring.

With a bit of organization though, we got things to work. Let’s see how we managed it.

Drawing of a calendar entitled "Manage It !". Collaboration and organization are key to succeed to use incremental software development for large scale refactorings.

This is the eighth post in a series about how to get sponsorship for large scale refactoring. If you haven’t, I encourage you to start from the beginning.

Manage it with a Design Vision

To succeed at anything, we need to know where we are going. It’s the same for large scale refactorings. We don’t need to have all the details of what we want to build. We do need a good enough draft to avoid going in the wrong direction. That’s even more true when we work as a team. Without a shared design vision, people will refactor in conflicting directions.

It’s very important to share the vision with all the team. We can stick high level UML sketches on the walls for example. As Kent Beck suggests, we can also use metaphores to communicate the design. In this talk, Nat Pryce explains that it’s a great way start, but that we will have to drop the metaphore later.


The idea is not to waste time in a Big Design Up Front. We just want to draft a vision :

  • We can run a Design level Event Storming (I guess I’ll have to blog about this some day)
  • We can grab a copy of Gamestorming and run another kind of collaborative design game
  • Or a few team members could work on something the way they prefer

Whatever the technique we start with, we’ll be able to refine and evolve the vision down the road.

Manage it with Mikado Graphs

Remember the ‘Mikado Method’ from my previous post ? It’s a technique to code and deploy large scale refactorings in baby steps. If you had a look at the reference links, you’ll might seen mentions of a ‘Mikado Graph’. Here is what it looks like :

Sample Mikado Graph. The Mikado graph is a great tool to manage and collaborate on incremental software development of large scale refactoring.

Sample mikado graph from http://mikadomethod.info/

As the number of refactoring steps grows, it becomes tricky to keep track of them all. A simple way is to draw them as nodes in a graph, and tackle the work from the leaves. If you are interested, check these posts about the Mikado Method. In my previous team, we became fans of the Mikado Method. We even built a tool to generate mikado graph from JIRA (our ticket management system) dependencies ! Using colors, we could track where we stood in the refactoring.

Sample Mikado Graph generated by our tool. Automating around Mikado graphs is of great help for large scale refactoring.

The key advantage of mikado graphs is that we can stick them on the wall. This enables all the team to know where we stand in a refactoring. This way, team members can collaborate during their slack. It can also make the boy scout rule more effective. When a developer happens to touch a file that appears in the graph, he or she can move it further in the good direction.

💡 A key benefit of mikado graphs is that we can stick them on the wall for everyone to know where we stand in a refactoring.

Manage it with Metrics

I mentioned coding conventions and a clear Definition of Done in a previous post.  Having code quality constraints is the compass of constant merciless refactoring. To make this actionable and real, we should take the time to setup an automatic metrics system. For example :

Doc Norton's talk about "The Technical Debt trap". His metrics are very useful to manage large scale refactoring.

  • Doc Norton suggests to track maintainability, coverage, complexity and coupling over time.
  • The A2DAM model suggests using specific rules to create Definition of Done constraints

Putting this in place will help everyone in the team to know if she or he should do more or less refactoring. The first benefit is that it prevents under and over engineering on new code. The second benefit is progress validation through metrics changes as we refactor.

Sample metrics dashboard. Out of the box, Code Climate supports 3 out of the 4 metrics suggested by Doc Norton. This can be really helpful to manage incremental software development.

Manage it with a bit of Planning

Granted, planning is not the most fun part of our job. It can save us a ton of work though. Joe Wright explains how they doubled their productivity by spending more time planning. If we want to make a good job of incremental refactoring, we’ll need to spend enough time preparing it. Important questions are :

  • What are the most important refactorings to work on ?
  • How many refactorings should we tackle at the same time ?
  • Are we making good progress on our refactorings ?
  • Why is this refactoring not yielding any visible results through our metrics system ?
  • Are there any news that should change our plans ?
  • Are we doing enough refactoring to keep things under control ?
  • What are the next steps in these refactorings ?
  • etc

I’m not talking about a big 6 month planning but rather regular short planning sessions. In Scrum this kind of planning happens every sprint. To make plannings more visual, engaging and fun, we might us something like Story Mapping. (I guess I should blog about this someday.)

💡 Keeping a Work In Progress limit on refactorings is essential.

Manage it through Time-Boxing

One last advice before I’m done. We must be very careful to time-box our work on refactoring increments. It’s all too easy, to get caught up in a refactoring during the slack at the end of the iteration. If we let the refactoring spill on features we risk loosing the business people’s trust.

Here again, using extra small baby steps helps to pause the refactoring. Another way is to do Kanban) style slack. Replace end of sprint slack by a fixed number of people slacking all the time. But I’ll come back to this in more details in a future post.

Next post

Using this set of practices my team was able to deliver large scale refactorings in small steps. That said, some refactorings are very difficult to technically deliver incrementally. Fortunately, people have come up with patterns like the Strangler and the Bubble Context to cope with this. That’s what I’ll go over in the next post.

This was the eighth post in a series about how to get sponsorship for large scale refactoring. I’ve already been through why it’s so difficult to get sponsorship for a refactoring. Why a badass developer attitude is important ? How to deliver refactorings steps by steps ? If you haven’t start by the beginning !

Incremental Software Development Strategies for Large Scale Refactoring #2 : Baby Steps

How can we exploit short time slots here and there to perform large scale refactoring?

My previous post was about how to get slots of time in your daily (or weekly) work to do some refactoring. I left my readers with the promise of techniques to fit the refactoring work into these small slots.

Obviously, it won’t be possible to perform any refactoring of any size in this way. With a bit of discipline and know-how though, it is possible to deal with quite a lot by splitting them up.

Baby steps are small increments of working software. The idea is that we test, commit, integrate and even deploy every small code change ! Using baby steps, we can perform large scale refactorings little by little. Refactoring in this way might seem involved, but it’s so much safer that it’s a no brainer once you’ve tried it ! Refactoring in baby steps can be challenging to master though.

Baby footprints. Taking really small baby steps when going through a large scale refactoring is safer

10 years ago, I used to work in a large bank in Paris. I had been dabbling on my own with eXtreme Programming for a few years, when we started a small project. I was to become the informal XP coach. The project was about connecting to an online brokering service. It involved adapting an existing domain library. It went out pretty well. More precisely, we created very few bugs in production, which was very different from the norm at the time. I remember this feedback from the manager :

We managed to move the code from X to Y through a succession of working baby steps ! That’s pretty uncommon ! A manager in 2006

Keep in mind that this was 10 years ago. We had not done anything special except trying to apply eXtreme Programming. Nowadays, as Continuous Integration has become mainstream these skills are becoming more common. That said, we still need to learn how to apply incremental software development to large scale refactoring. This is what I’m going to write about today.

This is the seventh post in a series about how to get sponsorship for large scale refactoring. If you haven’t, I encourage you to start from the beginning.

Team TDD Coding Dojos

Learning to work in baby steps is not as complicated as it might first seem. The safest and easiest way is to setup a team TDD coding dojo. With its Red-Green-Refactor loop TDD sets up a baby steps rhythm. As I’ll explain in my next post, baby steps work best when all the team uses them. That’s another thing the team Coding Dojo helps with.

💡 TDD has a baby steps rhythm baked in.

We can push the learning further. For example, we can use the baby steps constraint during a few coding dojo sessions. With this constraint, we revert the code if tests fail for more than 2 minutes ! Here is a way to go at it :

  1. Setup continuous testing : NCrunch in .Net, Guard in Ruby or Infinitest in Java)
  2. Only use automated refactorings or extremely simple code changes in order to ….
  3. … keep the code compiling all the time …
  4. … and cut the time the tests fail as much as possible

Mikado Method

One way to keep the tests green all the time is to use a slightly different TDD loop, as Nat Pryce suggests :

The red-green-refactor loop of TDD with an extra green arrow from failing test to refactor. This alternate TDD loop illustrates how to take baby steps with the Mikado Method


Here is how it goes. 

  1. Add a new failing test 
  2. If it’s trivial to fix, fix it. We are done
  3. If not, see what’s missing in the code
  4. Comment the test to get back to a green state
  5. Refactor the code to add what’s missing (and use other tests if needed)
  6. Uncomment the test
  7. Repeat from step 2

When doing this at the scale of a real life story or feature, we’d use git stash instead of comments. This way of working has a name, it’s called the Mikado Method. It is at the heart of making baby steps work in real life.

💡 The Mikado Method is at the heart of making baby steps work in real life

Take a break

With TDD and the Mikado Method we can put the refactoring on pause. We can perform a small increment of the refactoring, commit and deploy it … and pause ! We’ll work on business features for a while, and resume the refactoring later on.

A cup of coffee next to a computer. Developers can pause their large scale refactoring if they work in small enough baby steps

When done well, it feels slow. We have to remember that the alternative is to convince business people of prioritizing a refactoring … As we’ll regularly ship baby steps of a large scale refactoring, we’ll know we’re on the good track !

More to come

Unfortunately, even with bandwidth and skills, we are not there yet … It’s one thing for developers to do incremental software development of large scale refactoring on their own. It’s another to do it as a team !

This was the seventh post in a series about how to get sponsorship for large scale refactoring. Next post will be about how to manage constant merciless refactoring and baby steps as a team.

Incremental Software Development Strategies for Large Scale Refactoring #1 : Constant Merciless Refactoring

Here’s everything you need to find the time for constant merciless refactoring … without asking for the permission!

My previous post advocated incremental software development for large scale refactorings. It’s less risky and it prevents tunnel effects. Most of all, it’s a lot easier to convince business people of engaging in a refactoring this way.

It’s one thing to understand why it’s the way to go, but it’s another to be able to do it ! In this post, I’ll start by explaining how to find the time to do constant merciless refactoring.

This is the sixth post in a series about how to get sponsorship for large scale refactoring. If you haven’t, I encourage you to start from the beginning.

Constant Merciless Refactoring illustrated as a recurring cleanup activity

Steal Take the time for constant merciless refactoring

If it hurts, do it more often ! Wisdom of the internet

As a child, I used to be a very untidy kid. Every few week, my room would get in a real mess, and my mum would order me to clean all this mess. I would then lose 1 or 2 hours tidying my room up. She used to tell me that if I kept things tidy as I used them, I would not have to lose this time. From the top of my 10 years old, I would judge this advice as nonsense. 

This is the kind of mess my bedroom used to be in, when I was a kid, before I learned the virtues of constant merciless refactoring

Fast forward a few years, I am myself a parent and I’ve been working with legacy code for many years. These years have taught me how much I was wrong …

💡 The easiest refactorings to negotiate are the ones we don’t have to talk about !

The more refactoring we embed in ‘Business As Usual’, the more we’ll do, and the less we’ll argue with the business. We can wonder if this is still ‘professional’ ? In The Art Of Agile Development, James Shore explains that managing technical debt is the key to long term productivity. (Details in the Risk Management section of his book). As developers, we are the only experts in the code, the responsibility to keep it clean falls on us.

Never ask the permission to do a good job ! Doc Norton

There’s more to constant merciless refactoring ! It also keeps us in a good position to accept new features or refactorings later down the road.

Following are the 3 practices that make up constant merciless refactoring.

Yesterday’s weather and slack time

20 years ago, a promises of agile software development was to stick to a sustainable pace. When we are working with a flavor of Scrum, we can rely on it’s literature to inject some slack time.  Slack time is buffer time at the end of every iteration. We should not plan any work during the slack, to accommodate with the unexpected. It’s a way to deliver on forecasts, whatever happens.

In short, if your velocity chart looks something like that :

Drawing of a fluctuating team velocity. This is often the result of not enough constant merciless refactoring

Scrum tells us to plan what you delivered in your worst iteration for the next one ! When things will work bad, we’re still pretty likely to deliver what we wanted. When things work well, we’ll have time to tackle refactoring.

There’s a lot more to say about slack time. How to take some when you are using Kanban ? How to make sure you keep some when your velocity becomes pretty stable ? How to do you increase your velocity in the long term ? (I guess I’ll have to write a full blog post about this some day.)

The Boy Scout Rule

I already blogged about the Boy Scout Rule. Here is how Uncle Bob wrote it :

Always leave the file you are editing a little better than you found it. Bob Martin

Following this simple rule goes a long way to keep the code clean and ready for larger refactorings. It works arm in arm with Yesterday’s weather principle. The extra time we take for clean up impacts our capacity to plan stories and features. This creates time to keep on doing the boy scout rule in future iterations.

How ‘clean’ the code should be is a team decision. Coding conventions and a static code analyzer are very important to track the boy scout rule. I learned that code reviews, pairing, mobbing and coding dojos are great to agree on coding conventions.

Embedding refactoring in features

The Test Driven Development loop goes as Red-Green-Refactor.

TDD's Red-Green-Refactor loop. Itself highlighting Constant Merciless Refactoring as a recurring activity

The same loop goes on at larger scale for Acceptance or Feature Test. When repeated many times, the loop could as well be Refactor – Red – Green. In fact, it’s a lot easier to refactor when you know what feature you want to build than at the end of the previous one. (Nat Pryce wrote about that in more details)

💡 “Disguise” refactoring as first technical sub tasks of features to get them prioritized.

All this to say, we should start our features with the pre-requisite refactoring ! We should not build features on shaky foundations. We should also impact our estimates. In fact, it’s a lot easier to justify to business people at that moment. We don’t need to mention ‘refactoring’ or ‘clean up’. We can use technical sub-tasks to track these technical refactorings. Technical sub-tasks are the team’s and don’t need to be understandable by business people.

Technical tasks drawn "Under the sea" and visible to devs only. This leaves them room to do constant merciless refactoring

To be continued

Finding the time for constant merciless refactoring is one thing, but how do we fit the work in these short slots ? In the next post, I’ll continue about how to actually work in baby steps.

This was the sixth post about how to get sponsorship for large scale refactoring.

Incremental Software Development for Large Scale Refactorings

In the end, incremental software development techniques are almost always the safest way to refactor. Here is why.

My previous post was about the badass developer attitude. More specifically, how it can buy sponsorship for large scale refactorings. Unfortunately, attitude is not enough. We also need to be able to deliver in a way that builds trust with the business. Most of all, business is scared of the tunnel effect. Incremental software development techniques allows to deliver large scale refactoring step by step. Not only that, but it also allows to do so alongside business features. That’s how badass developers walk their talk and gain the business people’s trust.

This is the fifth post in a series about how to get sponsorship for large scale refactoring. If you haven’t, I encourage you to start from the beginning.

Drawing of a plant at different stages of growth, illustrating Incremental Software Development

Why does incremental software development matter ?

A short story

A few years ago, I joined a team whose work involved a Domain Specific Language and a parser. The parser had grown in an ad-hoc way for a few years, and was now both very brittle and difficult to extend. We knew the way to go was to adopt a more solid parsing approach. We wanted to migrate to ANTLR and a multi pass parser.

As always, the business was very pushy for new features. There was no way we could have stoped everything for a full parser re-write. We explained to them that some of their features would be impossible to write without the new parser. We suggested that we work on migrating the parser as a background technical Epic. We did so using incremental software development techniques.

It took us a few months to migrate the parser. Meanwhile, we kept the software in a releasable state. This allowed us to validate our progress on the refactoring. We could continue to release features. We were able to share our progress on the refactoring with  the business people. They were very happy with the way we did this refactoring. In fact, it set a standard about how to prepare the software for big features later on.

The real problems

To understand why incremental software development works, let’s understand the alternatives’ problems. The main alternative it to do the refactoring in one massive task. This kind of initiative screams “Tunnel effect waiting to happen” ! The tunnel effect scares business people for 3 reasons :

  1. Because they don’t know how much money they’ll need to put in the tunnel to get out of it
  2. Because they don’t know when they’ll get the other features which they are also waiting for
  3. To be blocked in the tunnel if something unexpected and important comes along

Picture of someone holding a light, alone in a tunnel. Incremental Software Development helps to avoid the tunnel effect when performing a large scale refactoring

Delivering a large scale refactoring with incremental software development fixes these 3 points.

  • Every commit is a step forward while keeping the system in a releasable state. If ever something unexpected comes along, we can pause the refactoring for a while. (point 3)
  • Not all the team has to work on refactoring at the same time. Working on the refactoring does not block the delivery of other features. (point 2)
  • Finally, after working on a refactoring for a while, it becomes easier to have an idea of how long it will take. (point 1)

💡 Incremental software development fixes the business people’s fear of refactoring tunnel.

It is true that performing the refactoring in one team-wide batch would be more efficient. It would reduce the overall Work In Progress and get it done quicker. Unfortunately, it’s also a lot more scary for business people !

Incremental Software Development techniques

Like any skills, we can learn these techniques. Some are easy, and we can learn them from books. Others are more difficult and are usually learned through hard won experience. They can also be learned from a veteran developer in your team who’s been through all this already. If you aren’t or don’t have a veteran around, we can still manage. Deliberate practice is a great way to learn almost anything. Coding dojos are the thing here (I’ll write more about this later).

Once we master these skills, a lot of things change. First, we can do refactoring without harming our relationship with business people. Second, it builds enough self confidence among developers to negotiate with business people. This in itself, makes us more credible in their eyes. As a result, they are more likely to compromise on prioritizing refactoring.

💡 Mastering incremental software development builds self-confidence for developers.

To be continued

This was the fifth post about how to get sponsorship for large scale refactoring. In the next posts, I’ll deal headlong with the actual techniques. How to get bandwidth ? How to work in baby steps ? How to track the progress ? How to deal with the large scale ? Finally how to go further ? As you can see, there is still a lot to come, so stay tuned !

Principles That Will Make You Become a Badass Developer

There are some simple attitude principles that badass developers follow to gain the business people trust.

In my last post, I went over things we should avoid if we want to become badass developers. As I said though, this is far from enough. Once we’ve stoped losing trust from the business, it’s time to build some more ! This is the forth post in a series about how to get sponsorship for a large scale refactoring. If you haven’t, start reading from the beginning.

The badass developers gain the business’s trust by sneaking in as business partners. A good way to become one is to start acting like one ! Here are examples of principles for that.

Arm of a badass developer with the tatoo '> Badass Principles;'

Keeping the business best interests in mind

This is all about decision making. We should try to steer decisions according to the business. Whenever we talk with business people, we should stay aways from technical topics. I still remember my younger self explaining threading details to a trader … Most of all I remember the look on his face ! We should avoid technical bla bla, but we should be clear about the business consequences.

Honesty and Candor

When we don’t agree with something, we should say so. When we don’t understand something, we should ask the question. We need to stick to facts and assume everyone has the business’s best interests in mind. Candor is a way to get our opinions and questions through, without sounding rude or pushy. There’s a whole chapter about candor in Creativity.inc, the book about Pixar.

Cover of the Creativity.inc book. It contains lessons on Candor we should all read to become badass developers

With time, business people will think of us as a positive and pragmatic problem solvers. That is exactly the kind of people they want to work with !

💡 Candor is a way to get our opinions and questions through, without sounding rude or pushy.

Strong opinions, but weekly held

Jeff Atwood, already wrote about this. The idea is to fight for our opinions, but let them go without a fuss when we proved wrong. We know that we are all very often wrong. Only fools or self-centered people don’t admit this reality. Business people won’t trust us in either case. We need to show that we can go over our previous opinions. This grows our reputation of rational problem solver.

Acknowledging when we don’t know

The same logic goes with knowledge. None of us knows everything. We have to admit when we don’t know something and ask for help. This proves that we place the business’s speed over our personal ‘know-it-all’ reputation.

Here is a story that happened to me at my first job. I’m sure most developers go through it one day or another. I was assigned a new task involving technologies I did not know. I did not have the guts to state upfront that I would have to learn them. The result was that I sent 2 full weeks fiddling with this task to get something working. The more it went on, the more the product people were wondering why it was taking so long, and the more I got stressed !

Be bold and say No !

If we are sure something we should not do something, we need to say so. Badass developers are not afraid to say they won’t do it. Good software engineering requires merciless prioritization. If there are doubt about the value of doing something, it’s often a better idea to make sure before wasting time on it.

There are many ways to say ‘No’. Before giving a harsh ‘No’, we can try to challenge decisions. We can ask for clarifications and rationals through open questions. Very often, highlighting the risks makes people review their plans. As technical experts, we should also share as much of the consequences as possible.

In the end, badass developers are ready to leave a FUBARed situation. Great engineers don’t have troubles finding jobs these days … There’s no reason they should accept to be waisting their time.

💡 In the end, badass developers are ready to leave a FUBARed situation

What do to next ?

As we become badass developers, our reputation will grow. We’ll be in a better position to negotiate a large scale refactoring with the business. There’s a catch though : we’ll need to live up to our reputation ! Admitting that we are wrong 100% with candor will not make it ! 

When we manage to negotiate a large scale refactoring, the team will need to do a good job of it. This boils down to delivering it piece by piece, alongside features. This is exactly what my next post will be about.

This post was the forth post in a series about how to get sponsorship for a large scale refactoring.

5 Mistakes Badass Developers Never Do

Here is a one sentence summary of my previous post.

Badass developers negotiate large scale refactorings with the business better.

Unfortunately, not all of us are sitting next to a true badass developer … Hopefully, we can all become one ! Depending on our track record, it’s going to be more or less difficult, but with time and the good attitude, we can all do it. Becoming a badass developer is all about trustworthiness.

This post is the third in a series about how to get sponsorship for a large scale refactoring. If you haven’t, I recommend you to start from the beginning.

The first thing to become trustworthy is to avoid things that kill trust. Sounds obvious, but it’s very easy to forget. Here are 5 examples of trust killers you should never do if you want to become a badass developer.

Drawing of a hurt finger after someone made a mistake with a hammer. Badass developer don't do this kind of mistakes !

Resume Driven Development

We should pick the best tools for the job. The best tools are often a bit old, precisely because they’ve been battle tested in production for a while ! That’s exactly the kind of technologies you want your business to rely on.

To keep his skills up to date, a badass developer will not add a funky new tech in the production code. He would rather negotiate slack time with the business. He might also openly take his Friday afternoons to experiment the latest techs ! He would simply explain that it’s to avoid polluting the production system.

💡 A badass developer will not add a funky new tech in the production code.

Over-engineering

Gold plating or over-engineering are similar anti-patterns. A badass developer always keeps the business’s interest in mind. This means he knows how to balance long term design and short term features. A good rule of thumb is to keep a vision in sight, but to get there in baby steps.

Build features with no agreed upon value

Product managers are here to select what should and what should not be in the product. As product experts, they are the ones who know how much a feature is worth. Except (maybe) when we are building tools for others developers, they know better than us. Adding something of dubious value in the product is bad in two ways. 

  • First, it takes some time to build, time that we could use to build valuable features instead. Remember : Real developers ship !
  • Second, it creates unnecessary code to maintain. In the worst case, it can constraint the architecture. Which might eventually prevent us from adding other more valuable features afterwards.

Hide in a tunnel

We should always be careful of the tunnel effect. Seeing their money vanishing with no visible output makes business people, understandably, creepy. As soon as things become too complicated, a badass developer will raise the alarm. The fact is that he has been in this kind of situation before, and knows how to recognize it. At that point, it’s still possible to discuss the problem with everyone, and adjust the plan.

As an interesting side note, I was at the Paris DDD Meetup last Thursday. We had the chance to welcome Eric Evans as a surprise guest ! When asked what were his worst mistakes, he said something along this line :

💡 Some of my biggest mistakes were not backtracking soon enough a few times as I was drifting in quagmire. Eric Evans

Eric Evans, the father of DDD, a true badass developer, answering questions at the Paris DDD meetup


Let the team down

It’s not only about getting the business’s trust. We must also build trust from our fellow developers. Whenever we break the build and leave, or worse, deploy and leave, that trust is gone for a long while… We should not do that !

There’s more to a badass developer

I’m done with this list of things badass developers don’t do. Avoiding these is only the first step to become a badass developer. In next post, I’ll write about what we need to do if we want to build strong trust with the business.

This post is the third post in a series about how to get sponsorship for a large scale refactoring.

Why We Need Badass Developers to Perform Large Scale Refactorings

My last post was about the challenge for dev teams to get sponsorship for large scale refactorings. I listed two main reasons :

  1. The business doubts developers to have their interests in mind
  2. They are also not aware of the cost of the current technical debt

This post (and the next) will be about how to gain the business’s trust. This is exactly where badass developers can help. Let me start with a story.

Drawing of 2 hands of a badass developer over his keyboard, with ">badass<" tatooed on his fingers

Back in 2002, at my first job, I started to read Refactoring, Improving the Design of Existing code. That’s where I read about unit testing. I found it so great that I made a demo to other programmers. Being the junior dev in the team, my co-workers reaction was something between “Meh” and “Maybe”. Fortunately, a more experienced and respected developer gave it a try. A few weeks after my demo, he announced to the team that unit testing worked great on new code. This time, people showed no questioning about his opinion : he if said so, it must have been true. Even business people blessed the practice !

The "Refactoring, improving the design of existing code" cover. Badass developers know how to perform large scale refactoring

I had given a good live coding demo, but it was this respected developer’s opinion that won the point. To convince business people of sponsoring a large scale refactoring, we need their trust. That’s why we need badass developers around.

💡 I had given a good live coding demo, but it was this respected developer’s opinion that won the point.

What is a badass developer

Badass Developer's fist with a ring "I am badass"

By Brooke LarkCC0, via Wikimedia Commons

Badass developers are first of all people who are credible to the business. This usually implies a track record of delivering features and refactorings. Badass developers understand the business constraints. That’s why they learned how to deliver refactorings alongside features. They also need to be responsible and bold enough to stand ground in front of the business. Finally, badass developers are able to train others.

💡 Badass developers are first of all people who are credible to the business

Unfortunately, there are not so many badass developers in the industry … It has a youngster bias, and tends to push experienced developers to other activities. As if 10 years of systems design was less valuable than knowing the latest language !

Learn more about Badasss developers

I tried to find other words before resorting to ‘Badass’. Unfortunately, I could find none that got the point so clearly. Uncle bob calls them ‘software professionals’ in The Clean Coder. Professionalism is not specific enough to me. Adam Nowak also calls them ‘responsible developers’ in a blog post.  That does not convey the idea that, sometimes devs need to stand guard in front of the business.

The clean coder book cover. Clean coder looks like a form of badass developer

These concepts, though, are very close to my definition of a badass developer. Check by yourself :

To be continued

This was why badass developers matter to the success of large scale refactorings. This was the second post in a series about how to get sponsorship for a large scale refactoring.  In the next post, we’ll look at what we can do to all become Badass developers.