Why and How to Start a #ZeroBugs Policy - Part 2

This is the second post of a series about the #ZeroBugs policy. In the previous post, I detailed what a #ZeroBugs policy is and how we tried to apply it. I encourage you start there.

Here is a quick summary of where we stood. We had tried to apply a #ZeroBugs policy with exploratory testing. We had discovered a lot of bugs, and we had agreed to tackle them little by little. We hoped to finish in a few months.

The Bug Panic game

The final bug rules

At about the same time, we started an improvement kata to increase our productivity. I wrote in details about the full story, but to make it short, here is what we discovered :

  • It would take us years, not months to fix all our bugs
  • Our definition for bugs was part of the problem

After some experimentation, here is the bug definition we came up with :

  • It corrupts data
  • It returns wrong results
  • It crashes with an unexpected error message

It stopped scope creep by removing the subjective “it used to work” rule. This rule also sped up bug fixing a lot. We closed many bugs with descriptive error messages.


Within 2 sprints, we reached a #ZeroBugs state. Backlog ‘management’ was simpler as it did not contain bugs anymore. We were also spending less time fixing bugs and more building stories. Even as we resumed our intense exploratory testing sessions, our bug backlog did not grow.

After a while, we discovered other benefits to the #ZeroBugs policy.

Spending less time on bugs made our velocity more stable. Our product owner was happier to see us deliver new features more reliably. In the end, this improved our relationship with him. 

💡 #ZeroBugs policy makes developers happier

It also made the team happier. First, because we had a lot less of stressful emergencies to answer to. Second, because programming new features is more interesting than fixing bugs.

A relaxed developer

It also made the other teams happier and more productive ! They encountered a lot less bugs when using our product. When they had problems, they usually got a clear error message. They would fix it on their side, or suggest us a new feature.

A #ZeroBugs policy and exploratory testing also have a side effect on technical debt ! Bugs tend to appear in the more brittle parts of the code. When fixing bugs, you took the opportunity to refactor technical debt at the same time.

💡 #ZeroBugs policy reduces technical debt

Finally, having a definition for bugs was very helpful. It allowed decentralized prioritization of bugs. The alternative to that is to have a domain expert available to sort bugs from stories. A decentralized rule prevents all sort of coordination and people issues. Our system was rather technical, which made it easy to define what a bug is. In other cases, it might be worth spending some time with the domain expert to define similar rules.

“Use decision rules to decentralize economic control.” From Principles of Product Development Flow by Donald G. Reinertsen

To summarize, sticking to #ZeroBugs is a win-win and sustainable situation for all.

Next week

This was part 2 of a series of posts about the #ZeroBugs policy. In next week’s post, I’ll provide a step by step guide about how to inject a #ZeroBugs policy in your team.

Why and How to Start a #ZeroBugs Policy - Part 1

Some teams spend 95% of their development time fixing bugs … An entrepreneur I worked with reported an even scarier story. He went bankrupt because bugs were taking all the development time. He had no time left for new features !

At the other end of the spectrum, some eXtreme Programming teams reported bug rates as low as 1.5 per month !

💡 Some XP teams reported bug rates as low as 1.5 per month !

Wouldn’t it be great for everyone if we had less bugs to fix ? Programmers would do more interesting work. Business people would get more valuable features. People would do less overtime. All this while users get a better product !

A bug in a viewfinder

Last year, in my team, we reached and maintained a #ZeroBugs state. Here is how we did it.

What’s the #ZeroBugs policy

Before we jump into our story, let’s take a second to understand what this is all about.

Bugs waste time. They waste the time of users, who can’t do what they need. They waste everyone’s time with interruptions from unhappy user. They steal time from new features development. If you have a large number of known bugs, you’ll also waste some time organizing them : eliminating duplicates, prioritizing them, ignoring others …

A #ZeroBugs policy is the convention of having no bugs in your backlog. This means that when someone finds a defect, it should be immediately :

  • fixed before any other feature
  • prioritized as a new story
  • or ignored (for the moment)

Obviously, the less bug you create the easier this will be.

Our journey to a #ZeroBugs policy


We were an R&D team, building a data management system. Our product was moving outside of pure research and development. Other teams were starting to use it and often reported bugs back to us.

We’d first read about #ZeroBugs in The Art of Agile Software Development. Given enough automated testing, XP teams could almost stop doing bug fixing !

Cover of the book "The Art of Agile Software Development"

As we had very solid automated testing in place, we decided to take the extra steps to save time on bug fixing.

First attempt

There is a straightforward way to install a #ZeroBugs policy. Make sure to have a customer or a Product Owner to prioritize defects as soon as they appear (check this story for an example).

Unfortunately, this would not work for us. Our Product Owner was great at providing us with a long term vision. He could not be available for day to day bug prioritization though.

Our first try was to propose a definition for bugs that we could use ourselves to classify them. We decided that something was a bug if either of the following was true :

  • It corrupts data.
  • It returns wrong results. As we were building a data management system, this was rather easy to verify.
  • It used to work but is now broken

Whenever we found an issue meeting any of these points, we would start working on it straightaway.

💡 Exploratory testing goes hand in hand with #ZeroBugs policy

To speed up downstream integration of our product by other teams, we also started Exploratory Testing. Exploratory testing is a practice that goes hand in hand with a #ZeroBugs policy.

Unearthing a bug debt

During the first exploratory testing session, we found more than 10 bugs. We thought that with a bit of focus we could fix all lingering bugs in a few weeks max … After 2 other sessions and more than one sprint of intense bug fixing, we had even unearthed more bugs. We were wrong, this would take a lot more than a few weeks.

If we kept on our 1 hour exploratory testing sessions every sprint, we would fill our backlog with bugs. We would also be fixing bugs for a few months before we could tackle any new feature !

We agreed on the following strategy to fix all these bugs little by little :

  • stop these intense exploratory testing sessions
  • do shorter exploratory testing sessions at the end of every story
  • fix 5 to 6 bugs every sprint

We hoped to fix all our bugs in a few months.

Next week

That was the first part of a series of posts about #ZeroBugs policy. In the next post, I’ll explain how we dealt with this bug debt and dig into the consequences it had on our work.

Sustainable Productivity in eXtreme Programming

eXtreme programming will not improve your short term productivity. But it will drastically improve your long term productivity.

My last post detailed how prioritization and Work In Progress limits are the keys to less work and higher productivity.

XP is an experiment in answer to the question, “How would you program if you had enough time?” Kent Beck

Cost of changes over time, for traditional and XP ways of building software

The best jam is made in old jars

As I was writing the post, I could not stop thinking :

Once again, XP has been addressing these WIP and prioritization issues for 20 years now !

Let’s see exactly how.

WIP limitation

XP has a drastic way to reduce WIP by 2 : pair programming. Instead of 6 developers taking on 6 stories at the same time, they only tackle 3. I won’t dive into the ton of other advantages to pair programming here. If you want to learn more, these blog posts might help.

Although it is now seen as a Scrum practice, Planning Game was part of original XP. Sticking to a basic planning game will give room for slack at the end of iterations. The trick is to leave the infamous focus factor away.

💡 A simple planning game will give room for slack.

Suppose a team member takes vacations for a sprint. The team should deliver less user stories that sprint. They’ll have less story points to schedule on next sprint. This leaves room for slack and long term improvements such as refactoring, learning … If half the team goes on holiday, that’s a different story, adapt your schedule with gut feeling.

A funny aspect of this is that XP teams are now the most likely to switch to the more extreme #NoEstimates.


XP’s on-site customer is a straightforward way to work only on the most valuable features. Who better than an end user would be able to make this estimation ?

The YAGNI (You Ain’t Gonna Need It) and simple design principles avoid building unnecessary features. TDD’s strict point of not writing code before a failing test prevents over-engineering.

Modern XP

In this wonderful talk from Rachel C. Davies, she presents how her team has been improving XP for 15 years. She explains how they organize as 2 mobs and 1 solo. At any moment, one mob is working on stories. The other mob works on important long term technical improvements. The solo does some learning. This is an even greater WIP reduction technique than pair programming.

Having a dev doing learning all the time is a form of continuous slack. If the team feels that they cannot deliver something by an important date, he can join the mob to help them.

There’s even more to this learning. It can be about anything. Developers work on their technical skills, but they also learn about the domain. Little by little this turns them into domain experts. Guess what : domain experts are great at optimizing value !

💡 With time and efforts, developers can become domain experts !


Unfortunately, there is no easy way to prove the long term productivity of XP. We cannot split the world to run the same project with and without XP. Another difficulty is that after a years of XP, work remains smooth and sustainable. Nothing like crunch mode. People used to the hero culture are hard to convince that they can be more productive by doing less.

Hopefully, lean theory and gut feeling of programmers who have switched to XP can back up my claims.

Are Software Developers Overworked or Undecided ?

During the last 15 years, I’ve worked in many different teams. Every team has its own way of working. I’ve been at places where everyone seemed busy all the time. Did we deliver more ? No. Often quite the opposite ! The most productive teams I’ve been in used to do no more than 6 hours of solid work every day …

A developer in a mouse wheel

What’s the reason behind that ?

In Lean Management theory, lead time is the time it takes to get a feature from customer request to delivery. Here is a graph that draws typical lead time over factory load (1 – idle time percentage).

A graph of the lead time versus utilization

What does all this tell us ? 

When utilization becomes too high, people start multitasking. As multitasking increases, productivity drops, delaying projects. As a consequence, products are late to hit the market. On the other side, a low utilization means paying people to be idle. That might put any company out of business. Factory managers’ best practice is to limit Work In Progress by loading their assembly lines around 80%. Not too high to avoid over-utilization, but not too low to avoid wasting time.

Every software product is new product development (Craig Larman). As such, they are way less predictable than any assembly line. It means we need more slack to absorb unexpected variations (Donald G. Reinertsen). This tells us that we should have more than 20% of slack in our schedules and plannings. Unfortunately, overloading and multitasking is a wide spread disease in the software world.

How to get rid of task switching ?

The answer is obvious : prioritization ! Only work on the most important item at a time. It’s not because we start everything now that they’ll sell them earlier ! It’s quite the opposite in fact !

A comparision of sequential and concurrent product development

Here is another advantage to prioritization. It decreases the time between start and completion of a feature. This, in turn, decreases the likelihood of building something obsolete.

One last thing about prioritization …

A graph of values of features in decreasing order

“Black Swan Farming Using Cost of Delay” Joshua J.Arnold and Özlem Yüce

This graph tells us that a small fraction of the features bring most of the value, while a very large portion of the features bring almost no value … As puzzling as this graph may be, it’s also a wonderful opportunity. If we manage to focus one the most valuable features only, we can save a ton of work, while being more effective ! Lean startup and market research are techniques to identify these features up-front.

💡 You don’t need more time, you just need to decide ! Seth Godin

Culture issue

Becoming more effective by banning overtime, enforcing monotasking and being serious about prioritizing ? These ideas are heresy in some workplaces ! Have a look around you. Are people overworked ? Are they fire fighting all the time ? Are they drowning in multitasking ? If it’s the case, I bet there’s a wide scale lack of prioritization issue at your workplace.

People at the top of the company need to control the overall WIP. If they don’t, multitasking will catch you up, however you try to escape it. Multitasking can be obvious e.g., when we need to work on different projects at the same time. It can also sneak in as maintenance, or unrelated meetings marathons. Another symptom is when developers need to take on “submarine” refactoring initiatives. From experience, that’s often the case in sales driven companies. Where any customer request is agreed and sold, regardless of the company’s capacity.

💡 Require​ ​everyone​ ​in​ ​product​ ​and​ ​engineering​ ​to​ ​read​ ​​Principles​ ​of​ ​Product​ ​Development​ ​Flow by Donald Reinertsen. John Cutler

There’s an interesting parallel in the Stephen R. Covey’s 7 Habits of Highly Effective People. He says we should drop the urgent and not important for the important and not urgent. In software term, it means spending time on important and long term work rather than short term stuff. We should do more refactoring, hiring, training and continuous improvement. We should stoping building features we are not sure of the value.

Let’s summarize. Here are my advices to work less and be more productive. Enforce hard WIP limits at all levels. Do serious prioritization. Provide enough slack. In my next post, I’ll show how XP addresses these issues.

Mobbing Around the World for 48h … Who’s in ?

In Far From the mobbing crowd the Cucumber guys explain how they combined mob programming and remote work. Matt and Steve also explain that a mob is both resilient and fast. They explain that they sometimes have to leave the mob for 15 minutes or so. When they come back the code has changed a lot ! It’s as if the mob continues no matter who’s in.

One of them then suggests an idea. 

With enough people around the world, we could run a mob that never stops!

A drawing of a mob walking around a binary world

What’s the point ?

In The Ultimate Guide To Remote Work the Zapier team explains how to take advantage of time zones and remote work. We can get a feature done faster by having a pair of programmers from different time zones work on it ! Instead of a single day of work in 24h, you get 2 ! We could get features in half the time.

The cover of The Ultimate Guide To Remote Work

In Lean Management, the time to get a feature is called the cycle time. A short cycle time has a lot of benefits. You get feedback faster. The needs of the users have less risk of changing. You’ll get less interruptions, which increases your focus and effectiveness.

Zapier found that time zones and remote work can divide the cycle time by 2. This is already a major improvement. Let’s replace this pair of programmers by a mob ! The cucumber guys explain that 4 hours of mobbing is about the most someone can do in a single day. With a mob of 5 people at any time, that’s around :

24/4 * 5 = 30

30 developers working on the same feature, full time !

What would be the effect of this on cycle time ? I don’t expect it to cut the cycle time by 30, but from my experience with pairing, I’d still expect a drastic reduction. Imagine a complex feature that you estimated to 30 days to finish. Could it be done in less than a week ?

Working on a single story at a time, we could forget merges and complex CIs ! More generally, it would lead to an important Work In Progress (WIP) reduction. Very related to cycle time, reducing the WIP also brings a lot of improvements. One of the most important is an increased throughput. Here is a great article detailing the benefits of reducing the WIP.

💡 Mob programming reduces WIP and cycle time

And what about the team structure ? Could a team of 30 be effective in this setting ? 7 is the magic count for a team size. If we could grow past this number, we could imagine completely new org charts :

  • More decentralized
  • More resilient 
  • Less synchronization and coordination overhead

A screen capture of 'Far from the mobbing crowd'

Let’s try it !

As any new idea, it’s also very likely to fail miserably … The communication overhead could turn out overwhelming. The mob might not be resilient to frequent changes after all. It could work but not bring any reduction to cycle time. There is a ton of reasons for this to fail.

The best way to find out is to try !

💡 Stupid ideas that work become great ideas. [XP moto]

We just need to find enough people spread around the world to give it a try. Could we do 48 hours straight of mobbing and go twice around the globe ?

We would need people distributed across all time zones. Pairing and incremental design would be de-facto practices. It would work better with people used to XP.

We’d need at least a mob of 3 doing 5h of mobbing per day for 2 days. That would be :

24/5*3 = 15

We could go with 15 people but 30 would be ideal. Once we have enough volunteers, we’ll have to agree on a few other things :

  • A date for the experiment
  • A subject we can all understand
  • Technologies and tools to use
  • A time-schedule

There’s nothing impossible here !

Call for volunteers

If you want to participate, leave a comment below or contact me through twitter. I’ll keep you updated about the status of the experiment.

One last thing … as you’ve guessed this is an experiment, not paid work. Anything we would produce will be open sourced on Github.

What Rails Teaches Us About Building Platforms and Frameworks

More than ever, the cheapest way to build a framework is to refactor it out of a specific app.

Rails is a web development server side framework built in Ruby. It’s been and remains very popular. It set the tone for all the MVC web frameworks that followed. People have ported it to many other languages. Rails is now in it’s 5th version, what is less known is how it was built in the first place.

Drawing of a train being detached from its rails

The story of Rails

David Heinemeier Hansson, the creator of Rails, is a cult in the Ruby community. You can find the history of Rails at many places. If you have a bit of time, read this great article from Wired. If you don’t have a lot of time, here is a summary.

Jason Fried and DHH were working at 37signals. They were working on project management app for small businesses called Basecamp. After releasing the first version of Basecamp, DHH extracted and open sourced Rails out of it.

37signals later then re-used Rails to build other apps like Campfire, Highrise and Backpack.

We all know the end of the story, Rails adoption exploded. Many successful companies like Github, Airbnb, Twitter & Shopify have used it. It evolved a lot, through the darwinism of open source. Today, Rails might not be the latest and coolest web framework, but it is still very productive and popular.

The Rails logo

Economic sense

To summarize, here is how they did it :

  1. Built a specific app, and monetized it
  2. Extracted an open source framework from it
  3. Built other apps on the framework

If you are not a programmers, you’re likely to assume that the cost of reusing software is negligible. If you are a programmer, I’ll ask you to assume that it is for a while.  Under this hypothesis, all they did makes a lot of business sense :

  • Building a specific app first is the fastest path to paying customers. Building a framework first would need more work.
  • Once you have paying customers, you’ve got money to fuel further work.
  • In particular, you have money to finance the extraction of an open source framework.
  • Putting the framework open source increases its reusability through bug reports and contributions.
  • Building the next app is even easier thanks to a mature framework and the revenues from the original app

💡 Once you have paying customers, you’ve got money to fuel further work.

How can it work ?

Cover of Rework book

Ok, enough for common sense. Let’s see what happens when you take into account the real cost of changing software ? All software developers know that changing software is far from cheap. Very often, it’s more expensive to adapt than to rebuild

So how did the Rails guys manage it ? Rails has 3 specificities that explain that :

  • Rails has automated testing built in and out
  • Basecamp guys are agile with a lower ‘a’. They don’t follow Scrum or any method. Read their books and you’ll understand how agile they are. They follow principles like KISS, YAGNI, Lean startup discovery …
  • Finally, it was open sourced !

Agile principles and technical practices is what allowed them to take the common-sense path.

💡 Thanks to agile practices, the cheapest way to build a framework is to extract it from a specific app.

Open question

If it is possible to do the thing that makes the most economic sense, why isn’t it the default way ? Why are so many of us still losing a ton of money writing large frameworks and platforms up-front ? How could we get large companies to adopt the more nimble ‘basecamp’ way ?

If you have remarks, answers or different point of views, I’d love to read your comment !

How to Run Your First Improvement Kata

The improvement kata can solve problems that typical retrospectives fail to address. Although there is a halo of mystery around it, it’s actually not that difficult to get started ! Here is a guide.

During the last few weeks, I’ve been blogging about the improvement kata. You can read the full story of the first time we applied it in our team to gain 25% of productivity. If you are more interested by what it taught us, check this other post.

Illustration of the improvement kata cookbook

A 5 bullets summary

Here is how I would explain what the improvement kata is :

  • It’s a continuous improvement technique. It relies on the scientific method to reach a target state.
  • It involves running experiments to know if your ideas are valid.
  • It can be long to run through, but it works on tricky situation where retrospectives don’t.
  • It’s 100% scientific. It uses data analysis and deduction. Not gut feeling and community best practices.
  • It can be part of the backlog, as any other item. It does not have to be a special event like retrospectives usually are.

The main steps of the improvement kata

💡 The improvement kata is 100% scientific, it uses data and deduction, not gut feeling and best practices

Let’s give it a try ! 

Is that enough for you to give it a try ? If so, great, read on ! If you need a bit more convincing, check the full story of how we gained 25% of productivity with it.

Here is how to get started :

  1. The first thing is to read about it. If you have the time, Toyota Kata is a good read. If you read French, the “Petit guide de management lean à l’usage des équipes agiles” is a very pleasant and easy read. Finally if you want to cut it as short as possible, read the Toyota Kata website.
  2. Pick a topic to try it on. Best candidates are clear and important problems. They might have emerged out of a retrospective for example. The scope should be small enough not to get lost.
  3. Once you’ve identified a topic, someone or a pair should take ownership of the kata. It’s very unlikely that you’ll be able to do the full kata in one afternoon. Understanding happens when the brain is at rest, and experiments take time. The owners need to dedicate some time to follow up on the kata.
  4. Repeatedly ask yourself the coaching kata questions. This will help you and your pair to stay on track.

    1. What’s the target condition ? (Describe what are you trying to achieve)
    2. What’s the actual condition ? (Describe your current situation)
    3. What obstacles do you think are preventing you from reaching the target condition ? Which one are you addressing now ? (Describe the first problem you are about to try to fix)
    4. What is your next step ? What do you expect ? (Describe the experiment you are going to run to test a solution)
    5. When can we go and see what we have learned from taking that step ? (Run the experiment and learn from the results. Decide on a process change or repeat from an earlier step)
  5. Going through the kata in pair is a great way to spread the practice within the team. At some point you might be able to run many improvement katas in parallel ! Just make sure not to walk on each other’s toes …

Expect the first time to be a bit rocky, and to feel lost from time to time …

Start today !

The main steps of the improvement kata

Many practices and techniques seem daunting at first. Remember the first time you wrote a test before the code. The first time you tried to program using only immutable data structures. Or the time you wrote your first “hello world” program !

💡 We can learn anything on our own by just doing it

The improvement kata is no different. Give it a go, and you’ll learn a powerful technique.

Whether you have already used the kata, you plan to use it or you have questions about it, I’d like to hear from you ! Leave a comment.

Lessons Learned From Running Our First Improvement Kata

During the past few weeks, I blogged the story of our first improvement kata.

Doing this first improvement kata taught us many lessons. We re-discovered best practices of the software industry. We also learned more general things about the improvement kata itself.

Drawing of books

Rediscovered best practices

As we went through the kata, we ‘proved’ many known best practices. We did not have to believe them, we had data explaining that they worked. For example :

We also pushed the #NoBug policy further than it is usually done. We defined a clear definition for bugs that anyone could use. Doing so, we removed the product owner (or on-site customer) from the picture. Very often, the PO is the only one who can sort stories from bugs out. We created what Donald Reinertsen calls a distributed rule in the flow book. It increased empowerment, removes bottlenecks, while ensuring alignment.

The 'Flow' book cover

The improvement kata

The first general lesson that we learned is that the improvement kata works !

At the beginning, we were very uneasy not to have perfect data. Remember how we had to resort to velocity as a proxy measure for productivity. In the end, that was not a severe problem. It still allowed us to understand what was going on.

We also learned that retrospective is not the only road to continuous improvement. In fact, the improvement kata and retrospectives are very different :

  • The time frame is different. A retrospective lasts for 1 or 2 hours and yield immediate results. An improvement kata is a background task that could take months in theory !
  • But the improvement kata also digs a lot deeper in the topic and brings true understanding. In our case, it fixed a problem that retrospectives where failing to address.
  • Ownership is also different. Retrospectives are a team activity. The improvement kata needs one or a few owners. It would be very difficult to align everyone on the same path of thoughts if we did it as a group activity.
  • Being a team activity, retrospectives have built-in alignment. The conclusions of the improvement kata must explained and agreed for a team to act on them. A good practice is to have regular (short) team meetings to share the current state of an improvement kata.
  • As the improvement kata is a more individual activity, it is more remote friendly. Team members can run the kata on their side, sharing everything through a wiki, or a blog for example.

Keep in mind that this was our first try at the kata. Some of our difficulties might disappear with a bit of practice !

What’s next ?

Hybrid continuous improvement

I clarified the differences between the improvement kata and retrospectives. That’s not the end of it. I’m sure a mixed format could be very interesting ! Start with a retrospective to collect the team’s problems, and vote on the more important. Add a corresponding improvement kata task to the backlog. Someone would then handle this improvement task, sharing with the team along the way.

This might be a great opportunity to reduce meeting time with shorter retrospectives.

💡 Reduce meeting time with mixed retrospective & improvement kata

Data science for software

Going through this improvement kata made me realize something else. It is very difficult to get quality data to investigate. We had to resort to what was available all the way.

What’s striking is that we use software tool for all our work ! These tools have logs and could record usage data somewhere. Imagine all we could learn by mining all this data ! Our IDEs, CI servers, quality monitors, test tools, version control and project management tools should store everything we do at the same place !

With all this data, could we improve our estimations ? Could we find creative ways to be more productive ? Could we estimate the speed up that fixing some technical debt would bring ?

💡 What are we waiting to apply data science to the development process ?

As the saying goes, “The cobbler’s children go barefoot”. We are building such systems for others, but not for ourselves.

Hopefully, new tools like CodeScene are emerging to fill this gap. You can learn more about CodeScene on their website, or from the founder’s book. It analyses the version control history to find hot spots and other things in your code.

The 'Code as a Crime Scene' book cover

While we keep dreaming of such tool, I’ll continue to blog. Next week, I will write a short guide of how to run your first improvement kata.

How We Used the Improvement Kata to Gain 25% of Productivity - Part 5

This is the fifth (and last) post of a series of 5 about the improvement kata. If you haven’t read the beginning of the story, I recommend you start from part 1.

In the previous post, we decided to adjust our definition of a bug to limit the time lost on nice-to-have bug fixes.

It would take a while to know if adjusting the definition of a bug would help us or not. At the same time, we knew it would not help us to reduce the number of bugs we escaped to other teams.

A 'SUCCESS' banner in the wind

Idea 3 : More exploratory testing

We decided to push on this matter as well. This means that we would be running two PDCAs (Plan-Do-Check-Act) at the same time. This is not the improvement kata procedure by the book. That could have been an error from our side, as first time users of the kata. Or maybe it’s a pragmatic habit to adapt the kata to real life … I guess we’ll know better as we apply the kata more often. The danger is that the different experiments can conflict with each other. When measuring the results, it becomes difficult to link observations with experiments. Anyway, back to our own situation, as you’ll see, it ended up well for us.

The first thing was to know a bit more about our bugs. Checking the recently closed bugs yielded suspicions about a recent features. Analyzing further proved our gut feeling.

Curve of how bugs were fixed on the last 2 months

Curve of how bugs were fixed on last 2 months

Curve of the origin of bugs on the last 2 months

Curve of the origin of bugs on the last 2 months

Ignoring the Christmas drop at the middle of the curve, we concluded 2 things from these graphs :

  • We were leaking bugs to the product
  • Bugs mostly came from newly added features

Despite all our automated tests and regular  exploratory testing, we were leaking bugs.

We decided to do more exploratory testing for a while ! We were already doing exploratory testing at the end of every story. We added an extra 1 hour team session of exploratory testing every sprint.

Do, Check & Act

We used these new conventions for a few weeks. We did more exploratory testing, and would be more strict about what a bug was. We stuck to our prioritization : first improvements, then bugs and only after, stories.

After a few weeks of that, we were able to update our bug trend and do a linear regression on it again. Here were the results :

Curve of the origin of bugs on the last 2 months

Hurray ! As you can see, we were to be done with bugs around April 2017, which was 3 months away at that time.

💡 Quality is free, but only for those willing to pay for it ! [Tom DeMarco in Peopleware]

Cover of the 'Peopleware' book by Tom DeMarco & Timothy Lister

We confidently adopted these practices as part of our working agreements. This brought our first improvement kata to its end.

💡 The improvement kata not only brings improvements, it also teaches you why they work.

3 months later

As you know, April 2017 is long gone. I can now give you a more up to date report of the actual effects on the team’s daily work. 

First, the backlog does not contain bugs anymore. We payed the bug debt back. Second, we still discover some bugs from time to time, but a lot less than we used to. To summarize, there is now a pair of developers (25%) of the team that can work on user stories instead of fixing bugs.

As we are still fixing bugs as they appear, the 25% productivity gain claim might be an overstatement, but 20% is not. At the same time, less bugs are now escaping. This means that the whole organization is saving on interruptions and rework. 25% might not be such a bold claim after all !

This is it !

This was post 5 in a series of 5 about the improvement kata. I’m not completely done writing about this improvement kata though. In the coming weeks, I’ll post about the lessons learned and how to start your own improvement kata.

How We Used the Improvement Kata to Gain 25% of Productivity - Part 4

This is the fourth post on a series of 5 about the improvement kata. If you haven’t read the beginning of the story, I recommend you start from part 1.

We ended the previous post with a target condition :

We’d like to be done with bugs within 3 months.

Currently, 1 pair (one quarter of the team) is constantly busy fixing bugs. If we manage to find a way to spend less time on bugs, we can expect a productivity increase of about 25%.

Anti bug insecticide can

Next step in the improvement kata is to run Plan-Do-Check-Act (PDCA) experiments. Before you run experiments, you need ideas !

Idea 1 : Stop exploratory testing

We’d like to spend less time fixing bugs. At the same time, we know that we started to spend more time on bug fixing when we started exploratory testing.

We thought that one easy way to do less bug fixing was to stop exploratory testing altogether ! We listed the pros and the cons of exploratory testing.

The obvious cons

  • It discovers bugs. Fixing these bugs costs time. We’d rather spend this time delivering new features

But the pros are many

  • It helps us discover technical debt and improve the quality of our code, which makes us faster in the long run. Clean code has less bugs. When we discover many bugs related to a particular region of the code, it means we should refactor it.
  • It speeds up integration with other teams’ work in many ways :
    • It saves other teams from debugging their time away trying to use broken things
    • Fixing bugs is sometimes just a matter of providing clear ‘not supported yet’ errors. Doing this saves a tremendous amount of time to other teams.
    • Avoids blocking other teams as they wait for your bug fixes
    • Reduces interruptions from bug reports and fixes bouncing between teams.
    • By reducing unplanned rework, it makes any commitment you do more likely

In the light of all this, stoping exploratory testing did not seem like a great idea. We had to find something else.

💡Saving time by not fixing bugs might not be a great idea

Exploration map with a red cross

Idea 2 : Review what a bug is

We needed to find a middle ground between where we were and stoping exploratory testing. We wanted :

  • not to let bugs escape
  • raise clean errors on things that are not yet supported
  • prevent scope creep from bug reports

A few months ago, when we decided to prioritize bugs before features, we had defined a definition for bugs. Here is what it looked like. Something is a bug if any one of these is true :

  1. It used to work
  2. It corrupts data
  3. It silently returns wrong results
  4. It returns an internal error (like an internal exception stack trace)

We examined this definition, discussed, and decided to dump the first clause. Our logic was that it was very difficult to know if something had already been working. The best thing we could refer to was our extensive suite of tests. If we discover that something is not working, it is very likely that it has never been. If it had been working, it means there was a hole in our test suite.

💡A hole in a test suite is a bug to fix

We decided to test this new bug definition for a while.

Not there yet !

We’re getting closer to the end, but you’re not done with PDCA yet.

This was post 4 in a series of 5 about the improvement kata. In the next and last post of this story you’ll learn how we ended the PDCAs for great results.