“Our world is increasingly complex, often chaotic, and always fast-flowing. This makes forecasting something between tremendously difficult and actually impossible, with a strong shift toward the latter as timescales get longer.”
—Andrew McAfee, Author of Machine, Platform, Crowd: Harnessing Our Digital Future

Applying Lean Estimating and Forecasting in Cadence

This is article six in the SAFe for Government (S4G) series. Click here to view the S4G home page.

One of the most challenging transitions in adopting Lean-Agile in government is the shift to incremental flow-based models for cost estimation and schedule forecasting. Legacy practices grounded in traditional project management methods require comprehensive cost and schedule estimates up-front. Careful control mechanisms are put in place to detect any variance to the plan while the solution is being built. Any change then requires detailed justification, approvals, and change orders that can cause delays and cost overruns. To mitigate these change events and still make a profit, contractors have managed the risk this model incurs by padding their time and cost estimates, or capturing the shortfall in future change orders.

Lean-Agile estimation and forecasting practices recognize that technology development frequently involves unknowns that make precise estimates challenging. This is especially true when building solutions or capabilities that have never been built before, where the cone of uncertainty is very wide at the beginning.  When the effort is smaller and/or the work is well-known, estimation has less risk and variability. It can be developed accurately from historical data. Lean-Agile also provides for agency adaptability to respond to changing conditions that necessitate adjustments to the program vision, roadmap, and backlog.

This article describes SAFe Lean-Agile alternatives to traditional cost and schedule estimating practices. (Note: The previous articles in this series provide foundational concepts such as Epics and Value Streams, so be sure to read them before continuing with the sections below.)

Details

Government leaders have a fiduciary responsibility to properly manage and account for the expenditure of taxpayer funds in the execution of agency missions. For technology development and operations that frequently require the use of external contractors, public sector leaders have the added responsibility of ensuring that the government receives quality solutions and services at fair prices.

Building, enhancing, fielding, and supporting technology-based capabilities typically begins with a government cost estimate and schedule of the initiative. In the U.S., the independent government cost estimate (IGCE) is the common process used to generate estimates for proposed programs to obtain approvals and funding before contracts can be awarded [1], [2], [3]. In virtually all countries, government program managers and contracting officers are responsible and accountable for ensuring technology capabilities are delivered within time, cost, and scope constraints. This estimating process often uses historical rates and information from previous, similar initiatives. But detailed, bottoms-up task breakdowns can also be used to justify the level of effort required.

SAFe recognizes that enabling agencies to plan and to manage ongoing program execution in a Lean-Agile environment still requires estimating and forecasting. What’s needed is an approach that:

• Balances fiduciary governance requirements with the ability to adjust estimates against macro-level constraints
• Accounts for both learning and for changing conditions

This is key to agency agility.

The sections that follow provide guidance on how to balance these concerns.

Estimating Using Epics

Some government programs are relatively easy to estimate and forecast. Take the example of a mature business system that requires only minor enhancements and bug fixes. For ongoing budget requests and contracting actions, cost estimates based on the historical quantity and skill levels needed, combined with the projected period of performance, can yield reasonably accurate projections.

However, the most impactful government programs are far more complex, involving the creation of new software and hardware solutions to enable entirely new capabilities. Examples range from re-platforming and modernizing mission-critical software to systems that leverage cutting-edge technologies, such as artificial intelligence (AI), autonomous vehicles, and new-age composite materials. In other words, they haven’t been built before, which means even historical data can be illusionary.

Detailed up-front designs and proposals are often used to estimate these programs. However, these can create false precision and unwarranted confidence in solutions that have many unknowns. History is littered with failed ‘well-specified’ programs that were ultimately canceled without producing a working system after egregious time and cost overruns. A better way is needed.

The companion article, Transitioning from project to a lean flow of epics, describes an epic as a large initiative of sufficient size and scope to warrant analysis, ROI assessment, a lean business case, and approval. The largest development programs—such as new weapons systems, or the wholesale replacement of a legacy financial system with an ERP—are likely to be a collection of related epics aggregated for governance purposes under a unifying program. Once a solution has been decomposed into one or more epics, Lean-Agile estimation can be used to develop cost estimates for each, and the program as a whole.

Because epics capture large complex initiatives that often have lots of uncertainty, a common approach for Agile estimation is to decompose the larger system or subsystem into smaller pieces of functionality called Features (Figure 1) that can be thought through individually. In SAFe, features (and their companion technical ‘Enablers’) are what Agile Release Trains (ARTs)—the team-of-teams that build the system—deliver and aggregate into the final solution. Practitioners on the ART continuously improve their proficiency in estimating the level of effort required to deliver features in the common currency of story points. (Note that these are high-level, order-of-magnitude estimates for forecasting, not the more detailed estimates that ARTs will create during actual planning and execution).

As long as government programs and contractors follow the Lean-Agile principle of long-lived teams-of-teams, the historical patterns of story points consumed to build features can be used to derive a cost-per-story-point metric for each ART. Multiplied by the estimated story points per epic, this calculation is typically sufficient and reasonably accurate for such exercises as an ICGE or equivalent. Program leaders still use historical data from similar features and epics as the best reference for estimating new epics (a current best practice even in traditional estimation). The primary difference between this process and the legacy approach is that estimates are based on the decomposition of the functionality of the product. This is not a detailed breakdown and estimation of the work tasks or techniques such as source lines of code (SLOC). If the work is entirely new and has many unknowns, prototypes and spikes may be needed as precursors to gain the learning needed to formulate reasonable estimates.

While this technique of decomposition to support estimation remains in use by many programs, additional cost estimation techniques describing estimating MVP costs, as well as forecasting implementation costs using T-shirt sizing based on historical data, can be found in the Epic article.

Forecasting Deliverables from the Backlog

Government fiduciaries not only need estimates of cost; they also need to be able to forecast when new capabilities can be delivered. To build Lean-Agile forecasts in SAFe, program leaders will require:

1. Long-lived Agile teams-of-teams (ARTs) with stable, dedicated practitioners
2. A prioritized backlog of estimated Epics (and the domain knowledge required)
3. An understanding of the historical velocity and available capacity of each ART that could potentially work on each epic for the next several Program Increments (PIs)

With these three data points, leaders can formulate multiple “what if” scenarios to determine when each prioritized epic can be delivered.

Figure 2 provides a simple illustration of this process.

 

In this example, a government program has three ARTs (each ART represents 50-125 practitioners). Based on historical data of actual delivery capability, we know that ART 1 can deliver ~1200 story points per PI, ART 2 can deliver 1000, and ART 3 can deliver 650 (the differences are likely due to the varying number of personnel per ART). The highest priority epic has been estimated at 1000 story points. In the planning process, each ART must communicate how much of their total capacity per PI they can potentially dedicate to working on the top epic. If the skill sets in each ART are similar, there are many possibilities for how the work could be divided and thus when the epic could be delivered. For example, if ART 2 has the skills to work on epic 1 and can devote 100% of their capacity, optimistically, that epic could be delivered in the first PI, or more realistically in PI 2 to account for issues and unknowns. To increase the probability of delivery in PI 1, features could also be distributed among multiple ARTs. However, if the skill sets in each ART are more domain-specific, then there is less flexibility, and planning will be needed to manage potential specialization bottlenecks.

Illustrating this what-if exercise can also expose where additional capacity is needed through addition or reallocation. Based on the agency’s needs reflected in the backlog, adjustments were made to reallocate resources from ART 2 to the other ARTs in Figure 2, providing the necessary capacity for the highest priority epics. Making these types of adjustments in cadence, only at the PI boundaries wherever possible, is the best practice.

Additional techniques for estimating duration can be found in the Epic article.

The key to using this long-range planning tool while maintaining a Lean-Agile process model is to remember this artifact is a forecast, not a commitment. Over the life of the program, fact patterns and the dynamics of the environment will inevitably emerge, requiring an update to the forecast based on the best knowledge available. It’s essential that government fiduciaries understand and embrace this mindset and avoid driving program leaders back into false precision estimating practices that have proven to have a high risk of failure.

Moving Forward

The next key concept for adopting SAFe in Government is Modifying Acquisition Practices to Enable Lean-Agile Development and Operations.