Measure the Business Value of Your Spikes and Take High Payoff Risks (Lean Software Development Part 4)

Lately at work, we’ve unexpectedly been asked by other teams if they could use our product for something that we had not forseen. As we are not sure whether we’ll be able to tune our product to their needs, we are thinking about doing a short study to know the answer. This looks like a great opportunity to try out Cost of Delay analysis about uncertain tasks.

Unfortunately, I cannot write the details of what we are creating at work in this blog, so let’s assume that we are building a Todo List Software.

We have been targeting the enterprise market. Lately, we’ve seen some interest from individuals planning to use our todo list system for themselves at home.

For individuals, the system would need to be highly available and live 24/7 over the internet, latency will also be critical to retain customers, but the product could get a market share with a basic feature set.

On the other side, enterprise customers need advanced features, absolute data safety, but they can cope with nightly restarts of the server.

In order to know if we can make our todo list system available and fast enough for the individuals market, we are planning to conduct a pre-study, so as not to waste time on an unreachable goal. In XP terms, this is a spike, and it’s a bunch of experiments rather than a theoretical study.

When should we prioritize this spike ?

If we are using the Weighted Shortest Job First metric to prioritize our work, we need to estimate the cost of delay of a task to determine its priority. Hereafter I will explain how we could determine the value of this spike.

Computing the cost of delay

The strategy to compute the cost of delay for such a risk mitigation task is to compute the difference in cost of delays with or without doing it.

1. The products, the features, the MVP and the estimates

As I explained in a previous post, for usual features, cost of delay is equivalent to it’s value. Along with our gross estimates, here are the relative values that our product owner gave us for the different products we are envisioning.

Feature $ Enterprise $ Individuals Estimated work
Robustness 20* 20* 2
Availability 0 40* 2
Latency 0 40* 1
Durability 40* 13 2
Multi user lists 20* 8 2
Labels 20 13 2
Custom report tool 13 0 3
TOTAL Cost Of Delay of v1 80 100

Stared (*) features are required for the first version of the product. Features with a value of 0 are not required for the product. Eventually, unstared features with a non null business value would be great for a second release.

It seems that the individuals market is a greater opportunity, so it’s worth thinking about it. Unfortunately for the moment, we really don’t know if we’ll manage to get the high availability that is required for such a product.

The availability spike we are envisioning would take 1 unit of time.

2. Computing the cost of delay of this spike

The cost of delay of a task involving some uncertainty is the probabilistic expected value of its cost of delay. We estimate that we have 50% of chances of matching the availability required by individuals. It means that CoD of the spike = 50% * CoD if we match the latency + 50% CoD if we don’t match the availability.

2.a. The Cost of Delay if we get the availability

Let’s consider the future in which we’ll manage to reduce the latency. The cost of delay of a spike task is the difference in Cost with and without doing the spike, per relevent months.

2.a.i. The cost if we don’t do the spike

Unfortunately, at this point in this future, we don’t yet know that we’ll manage to get to the availability.

Feature $ Enterprise $ Individuals $ Expected Estimated work WSJF
Latency 0 40* 20 1 20
Durability 40* 13 26 2 13
Robustness 20* 20* 20 2 10
Availability 0 40* 20 2 10
Labels 20 13 17 2 9
Multi user lists 20* 8 14 2 7
Custom report tool 13 0 8 3 3

We’ll resort to WSJF to prioritize our work. Here is what we’ll be able to ship :

Product Delay CoD Cost
Individuals 7 100 700
Individuals Durability 7 13 91
Individuals Labels 9 13 117
Enterprise 11 80 880
Enterprise labels 11 20 220
Individuals Multi user lists 13 8 104
Enterprise Custom reports 16 13 208
2320

2.a.ii. The cost if we do the spike

In this case, we would start by the spike, and it would tell us that we can reach the individuals availability and so that we should go for this feature first. Here will be our planning

Feature $ Enterprise $ Individuals Estimated work Enterprise WSJF Individuals WSJF
Feasibility spike 1
Latency 0 40* 1 40
Availability 0 40* 2 20
Robustness 20* 20* 2 10 10
Durability 40* 13 2 20 7
Multi user lists 20* 8 2 10 4
Labels 20 13 2 10 7
Custom report tool 13 0 3 4

Here is how we will be able to ship :

Product Delay CoD Cost
Individuals 6 100 600
Individuals Durability 8 13 104
Individuals Multi user lists 10 8 80
Enterprise 10 80 800
Individuals Labels 12 13 156
Enterprise Labels 12 20 240
Enterprise Custom reports 15 13 195
2175

2.a.iii. Cost of delay of the spike if we reach the availability

By making the spike, we would save 2320 – 2175 = 145$

Without doing the spike, we would discover whether we would reach the availability when we try it, around time 7 (see 2.a.i).

So the cost of delay for the spike would be around 145/7 = 21 $/m

2.b. The Cost of Delay if we don’t get the availability

Let’s now consider the future in which we don’t manage to increase the availability.

Using the same logic as before, let’s now see what happens

2.b.i. The cost if we don’t do the spike

Unfortunately, at this point in this future, we don’t yet know that we’ll not manage to get to the availability.

Feature $ Enterprise $ Individuals $ Expected Estimated work WSJF
Latency 0 40* 20 1 20
Durability 40* 13 26 2 13
Robustness 20* 20* 20 2 10
Availability 0 40* 20 2 10
Multi user lists 20* 8 14 2 7
Labels 20 13 17 2 9
Custom report tool 13 0 8 3 3

When we’ll fail at the availability, we’ll switch multi user lists and labels to be able to ship to enterprises as quickly as possible. Here is what we’ll ship.

Product Delay CoD Cost
Enterprise 9 80 720
Enterprise Labels 11 20 220
Enterprise Custom reports 14 13 182
1122

2.b.ii. The cost if we do the spike

In this case, we would start by the spike, and it would tell us that we won’t match the availability required for individuals and so that that there’s no need to run after this now.

Feature $ Enterprise Estimated work WSJF
Feasibility spike 1
Durability 40* 2 13
Robustness 20* 2 10
Multi user lists 20* 2 7
Labels 20 2 9
Custom report tool 13 3 3

Here is how we will be able to ship :

Product Delay CoD Cost
Enterprise 7 80 560
Enterprise Labels 9 20 180
Enterprise Custom reports 12 13 156
896

2.b.iii. Cost of delay of the spike if we reach the availability

By making the spike, we would save 1122 – 896 = 226$

As before, without doing the spike, we would discover whether we would get the availability when we try it, around time 7.

So the cost of delay for the spike is around 226/7 = 32 $/m

2.c. Compute overall Cost of Delay of the Spike

Given that we estimate that there is a 50% chances of making the latency, the overall expected cost of delay is

50% * 21 + 50% * 32 = 26.5 $/m

Inject the spike in the backlog

With the Cost of Delay of the spike, we can compute it’s WSJF and prioritize it against other features.

Feature $ Enterprise $ Individuals Expected $ Estimated work WSJF
Feasibility Spike 26.5 1 26.5
Latency 0 40* 20 1 20
Durability 40* 13 26 2 13
Robustness 20* 20* 20 2 10
Availability 0 40* 20 2 10
Multi user lists 20* 8 14 2 7
Labels 20 13 17 2 9
Custom report tool 13 0 8 3 3

The spike comes at the top of our backlog. Which confirms our gut feeling.

Conclusion

Doing this long study confirmed classic rule of thumbs

  • Don’t develop many products at the same time
  • Do some Proof Of Concepts early before starting to work on uncertain features
  • Tackle the most risky features first

By improving the inputs, we could get more quality results :

  • If we had access to real sales or finance figures for the costs
  • If we did some sort of poker risk estimation instead of just guessing at 50% chances

Obviously, the analysis itself is not perfect, but it hints to the good choices. And as Don Reinertsen puts it, using an economical framework, the spread between people estimations goes down from 50:1 to 2:1 ! This seems a good alternative to the experience and gut feeling approach which :

  • can trigger heated unfounded discussions
  • often means high dependence on the intuition of a single individual

As everything is quantitative though, one could imagine that with other figures, we could have got to another conclusion, such as :

  • The spike is not worth doing (it costs more than it might save)
  • The spike can wait a bit

This was part 4 of my Lean Software Development Series. Part 3 was How to measure your speed with your business value, continue on Part 5 : What optimization should we work on ?.

Comments