Please, stop buying Story Points
October 1, 2021
October 1, 2021
Since the dawn of Agile, contracting officers have sensed a disconnect between traditional government contracts and Agile delivery methods.
A central tenet of Agile is – You can’t know all the requirements at the start of the project. In the Agile Manifesto, it’s conveyed as Responding to change over following a plan. This idea contradicts how traditional contracts are often set up, with vendors handcuffed to the Iron Triangle of cost, scope and schedule.
Over the years, a number of contracting approaches have emerged to resolve this conflict. Yet, one popular approach is doing more harm than good – points-based procurement, or buying Story Points.
A Story Point is a “unit of measure for expressing an estimate of the overall effort required to fully implement a piece of work.”
To define a Story Point, a team identifies a simple, straightforward work item that can be completed quickly, usually in less than a day. They call it 1 Story Point and use it as their basis for comparison moving forward. Every other work item is then sized relative to that original work item. A work item sized as 2 Story Points should be twice the effort of the 1-pointer.
When analyzing work items, teams consider the volume of work, complexity, and risk/uncertainty. All three of these factors drive the overall effort required to fully implement that work item.
Unlike the traditional estimating approach, where an architect might generate all the estimates, with Story Points, the team doing the work is the one estimating. Another key distinction is that Story Point estimation happens continuously, throughout the execution of work, instead of just once, at the start of a project.
Keep in mind, the basis for comparison (the 1-pointer) is relevant just for the specific team that analyzed that work item. Another team might look at that same work item and conclude that the effort is much greater. So, Story Point estimates are relative and subjective, factoring in the unique blend of experience and skills on that specific team.
<<< Start >>>
Story Point estimates are relative and subjective, factoring in the unique blend of experience and skills on that specific team.
<<< End >>>
The general idea of points-based procurement is to shift the focus from buying discrete requirements to buying capacity in the form of these Story Points.
In a typical points-based model, the government issues a solicitation and includes a sample work item with a designated Story Point value (e.g., 1 point). Vendors will respond and quote a price based on the level of effort required to deliver that 1-point story. The government now has a cost per Story Point for each vendor. They then issue Task Orders with a pre-defined volume of Story Points and use the vendor’s cost per Story Point to fix the price. For example, if a vendor has a price of $500 per Story Point and is awarded a Task Order worth 200 Story Points, they can invoice the customer $100,000 for the work.
<<< Start >>>
<<< End >>>
The intent of this approach is to abstract away the specific solution requirements and team composition, and instead focus only on the value that can be delivered. The problem is - Story Points do not equate to value.
There are a number of issues with the points-based procurement approach:
People conflate Story Points with value: More experienced teams tend to estimate work items as fewer Story Points than their junior counterparts. But a points-based procurement incentivizes the delivery of Story Points, not value. So, on paper, a junior team that delivers less value may appear to be more productive because they complete more Story Points.
Story Points are easy to manipulate: Because Story Points are subjective and relative (no connection to hours), they’re quite easy to manipulate. Teams with Story Point quotas (goals) will almost always hit them. Even if they must inflate their estimates to do so. And who’s to say that the estimates are inflated? Story Points are team-specific. There is no “wrong” Story Point estimate. From a contracts standpoint, where vendors are bidding against blocks of Story Points, this has huge implications.
Story Points don’t build ownership: With points-based procurement, it’s common for agencies to “pre-estimate” a project in Story Points. They then multiply that estimate against a cost per Story Point to derive a budget. This breaks a core tenet of effective cost estimating – The people doing the work estimate the work.
By force-feeding estimates to vendors, we miss out on the shared understanding gained through collaborative estimating. Instead, we immediately jump to an arbitrary and subjective Story Point estimate that does not reflect the skills and experience of the team building the solution.
This fosters a material lack of ownership of the estimates and commitment from the workers bound to them. This can lead to budget overruns and change orders for new work items as they are discovered.
Our federal partners are succeeding with Agile delivery using a wide range of unique contract approaches. Every contract is different and your mileage may vary. But one contracting approach that is proving quite effective is fixed-price capacity.
Under a fixed-price capacity contract, we shift from buying requirements, hours or points to buying teams. As one government CIO puts it, You contract for the factory, not the car.
In this model, the customer defines the size and composition of each Agile team, and the vendor provides a set price per team. The customer pays a fixed price for each team for a set period of time (e.g., two-week sprint). This structure offers both cost predictability and the flexibility needed to iterate toward the ideal solution.
<<< Start >>>
<<< End >>>
Periods of performance are typically shorter in this model, giving the government flexibility. Based on demonstrated value, they can increase funding, decrease funding, transition to another vendor or ramp down the contract.
Points-based procurement is flawed. To be clear, Story Points themselves can be quite effective when used as intended. There’s no harm in Agile teams leveraging Story Points to estimate and forecast work during execution; that’s what they were designed for. What they were not designed for is to be used as a tool or formula for bidding on contracts. Put simply, Story Points do not belong in the procurement process.