MGMT 620 Week 6 Project Evaluation Research Example

Evaluating Projects Before, During and After Delivery: Current Research, Best Practices and Three Cases

Abstract

This paper examines current research and practice in project evaluation and tests five best practices against three published cases. Research on project success shows that delivery measures of cost, time and quality capture only part of a project's value, that success unfolds across several dimensions and time frames, and that organizations learn little from projects unless learning is designed in. From this research, the paper derives five practices: define success at several levels before the project begins, evaluate during the project as well as after it, assign owners for benefits, connect evaluation to decisions, and build learning into each phase. The cases of the Denver International Airport automated baggage system, the 2013 launch of HealthCare.gov and the opening of Heathrow Terminal 5 show what happens when these practices are present or absent. The paper concludes that evaluation is most valuable when it is treated as a management discipline that informs decisions throughout a project, not as a report written after the outcome is known.

Introduction

Organizations spend heavily on projects and then often learn little about whether those projects worked. A project closes, its team disbands, and the question of whether it delivered value is left to anecdote. Project evaluation is the discipline that answers that question systematically, and it has grown in scope as research has shown how narrow the traditional measures are. Thomas (2015) presents evaluation as a structured activity that should be planned at the beginning of a project and continue through its life, with lessons captured and applied. This paper argues that evaluation works when it is planned from the start, repeated at defined points and tied to the decisions it is meant to inform. It reviews the research, sets out five best practices and tests them against three well-documented cases, two public and one private, in which evaluation either saved a project or arrived too late to help.

What Project Evaluation Means

Project evaluation can be divided by its timing and by its object. By timing, formative evaluation takes place while a project is underway, when its findings can still change the work, and summative evaluation takes place at or after completion to judge what was achieved. Both are needed, because a summative review can explain a failure but cannot prevent it, while formative reviews can steer a project but cannot judge its long-term value.

By object, evaluation can address different levels of performance. Zwikael and Smyrk (2012) argue that the conventional test of project performance, comparing results to plans for cost, time and scope, is incomplete and often irrelevant to the people who fund projects. Their alternative grades a project three times over: once for how well its managers delivered the outputs, again for the owners' success in converting those outputs into results, and a third time for whether the money spent was worth spending. A project can succeed at one level and fail at another, which means an evaluation must say which level it is judging. A team that delivered its outputs on time deserves credit for that even if the investment did not pay off, and an investment that paid off despite a late delivery should not be counted a failure.

Current Research on Project Evaluation

Three findings from the research shape current evaluation practice. The first is that success criteria must extend beyond delivery. Atkinson (1999) pointed out that budget and schedule targets are fixed at the moment a project knows least, and that quality is judged differently by each person who looks at it, and he called for measures of what the project actually does for the organization that paid for it and for the people it affects. Shenhar et al. (2001) found that success has several dimensions that matter at different times, from efficiency in the short run to customer impact, business success and preparation for the future over longer periods. Ika (2009), reviewing decades of project management research, traced a movement from narrow delivery measures toward broader, stakeholder-based definitions of success.

The second finding concerns benefits. Serra and Kunc (2015), surveying practitioners in three countries, found that benefits realization practices, such as identifying benefits, assigning their ownership and tracking them after delivery, went along with projects that added more strategic value to the business. The implication for evaluation is that benefits must be defined before the project begins if they are to be measured afterward.

The third finding concerns learning. Busby (1999) observed that post-project reviews pass knowledge around and fix mistaken beliefs, but that their diagnosis tends to be shallow and their remedies superficial. Williams (2008), surveying project managers and the research literature, concluded that teams built to dissolve, working on one-off and tangled tasks, are poorly placed to pass anything on, so organizations must deliberately build the structures that carry lessons forward. Evaluation that produces lessons no one acts on is, in effect, incomplete.

Limits of Current Practice

The research also explains why evaluation is often weaker in practice than in principle. Evaluations during a project compete for time with the work itself, and reviews after a project compete with the next assignment. Evaluators who are part of the project may be reluctant to report bad news to sponsors who chose its direction, while outside evaluators may lack the context to judge what they see. Benefits that appear years after delivery are hard to attribute to one project, because other changes happen at the same time. And public projects face an added constraint: political commitments to dates and budgets can make evaluation feel like a threat rather than a tool. None of these limits makes evaluation less necessary, but each suggests that evaluation must be designed into a project's governance, with time, independence and authority, if it is to work.

Best Practices for Project Evaluation

The research suggests five practices. First, define success at several levels before the project begins: delivery targets for the project team, outcome targets for the owners, and investment criteria for the sponsors. Second, evaluate during the project, not only after it, through phase reviews and independent assessments that can change the work while change is still affordable. Third, assign an owner for each benefit, usually in the operating organization, who is responsible for measuring it after the project team has gone. Fourth, connect each evaluation to a decision, such as continuing, redirecting or stopping the project, so that findings have somewhere to go. Fifth, build learning into each phase by holding reviews while teams are intact and giving each lesson an owner and a deadline. Thomas (2015) frames these as parts of a single evaluation process rather than separate activities, and the cases below show what happens when they are present or missing.

Case 1: The Denver International Airport Baggage System

Denver's new airport was designed around an automated baggage system of unprecedented scale, intended to move bags among all concourses in carts on a network of tracks. Montealegre and Keil (2000) studied how the city's leaders handled the project as it fell behind. Repeated tests failed, and the airport's opening was postponed several times. The study describes a process of de-escalation, which the authors present as rare and difficult because leaders who have publicly committed to a course of action tend to defend it: after months of escalating commitment, the city commissioned an independent assessment, reframed the problem, and decided to build a conventional baggage system as a backup so the airport could open. The airport opened in 1995, sixteen months late, with the automated system serving only part of the airport. De Neufville (1994), writing during the crisis, argued that the system's scale and complexity had been underestimated from the start.

Measured against the five practices, Denver's story is mixed. Success had been defined largely as the technical system working, not as the airport opening, which made it hard to see that the system was endangering the larger goal. Yet the independent assessment was a form of formative evaluation, and it was tied to a decision, which allowed the city to change course. The case shows that evaluation during a project can rescue an investment even after early warnings have been ignored.

Case 2: The Launch of HealthCare.gov

When HealthCare.gov opened on October 1, 2013, as the federal marketplace for health insurance, the site failed under demand, and only a small number of people were able to complete enrollment in its first days. The Government Accountability Office (2014) later reviewed the Centers for Medicare and Medicaid Services' management of the contracts that built the site and found ineffective planning and oversight practices, including starting development before requirements were stable and not acting effectively on signs of contractor performance problems. A team of outside specialists, working intensively over the following weeks, stabilized the site.

Against the best practices, the case shows the cost of evaluation that does not reach decisions. Warning signs existed before launch, but they did not lead to a decision to delay or to change the approach. Success had been defined by a fixed launch date that the agency treated as immovable, so evaluation could only confirm that the date was at risk, not change it. The agency's own leaders later acknowledged that the launch had gone badly, and the rescue that followed was itself an intense form of formative evaluation, measuring the site's performance daily and fixing the worst problems first, which suggests what earlier evaluation might have achieved.

Case 3: Heathrow Terminal 5

Heathrow's Terminal 5 is unusual because it contains both a success and a failure. As a construction project, it was widely regarded as a success. Davies et al. (2009) describe how the airport's owner acted as systems integrator, managing risk and coordinating a large network of suppliers under a governance model designed for the project, and how the terminal was delivered on time. Yet when the terminal opened in March 2008, its baggage handling and operations failed in the first days: flights were canceled and many bags did not travel with their passengers. A parliamentary committee examined the opening and pointed to problems including insufficient testing and staff preparation for live operations (House of Commons Transport Committee, 2008).

Measured against the best practices, Terminal 5 shows that success at one level does not guarantee success at another. The project management level, delivering the building, was evaluated and managed rigorously. The operational level, turning a finished building into a working terminal on the first day, received less attention. In Zwikael and Smyrk's terms, the project's managers succeeded while its owners struggled to realize the outcome. Defining success at several levels before construction, and evaluating operational readiness as carefully as construction progress, might have prevented the opening problems.

What the Cases Show

The three cases point to the same conclusion from different directions. In each, evaluation worked best where it was tied to a decision and repeated during the project: Denver's independent assessment led to a backup system, and the HealthCare.gov rescue measured performance daily. Where evaluation was absent or arrived late, problems that were knowable became crises: HealthCare.gov's warnings did not change its launch, and Terminal 5's operational readiness was not tested with the rigor applied to its construction. All three also show the importance of defining success at more than one level. Denver focused on a technical system, HealthCare.gov on a date, and Terminal 5 on construction, and in each case the narrower definition hid a risk to the broader goal.

A Biblical Perspective on Evaluation

Scripture models evaluation that is honest, specific and aimed at improvement. In the letters to the seven churches in Revelation, the risen Christ commends each church for what it has done well and then names what it must change, as when he tells the church at Ephesus that he knows its works and endurance but holds against it that it has abandoned the love it had at first (English Standard Version Bible, 2001/2016, Revelation 2:2-4). The pattern is a model for project evaluation: an honest account of strengths, a clear statement of what must change and a call to act. Paul also urged the Corinthians to examine themselves (English Standard Version Bible, 2001/2016, 2 Corinthians 13:5), a reminder that evaluation begins with the willingness to look honestly at one's own work. For Christian project leaders, evaluation is a form of stewardship, not an exercise in blame.

Conclusion

Research on project evaluation has moved from asking whether a project met its budget and schedule to asking whether it delivered value, at what level and for whom, and whether the organization learned from it. Five practices follow from that research: define success at several levels before starting, evaluate during the project, assign benefit owners, tie evaluation to decisions and build learning into each phase. The Denver, HealthCare.gov and Terminal 5 cases show that these practices are not academic. Where they were present, evaluation changed outcomes; where they were absent, knowable problems became public failures. The most important lesson for project managers is that evaluation is a management discipline exercised throughout a project, not a report written after the outcome is known.

References

Atkinson, R. (1999). Project management: Cost, time and quality, two best guesses and a phenomenon, its time to accept other success criteria. International Journal of Project Management, 17(6), 337-342. https://doi.org/10.1016/S0263-7863(98)00069-6

Busby, J. (1999). An assessment of post-project reviews. Project Management Journal, 30(3), 23-29. https://doi.org/10.1177/875697289903000304

Davies, A., Gann, D., & Douglas, T. (2009). Innovation in megaprojects: Systems integration at London Heathrow Terminal 5. California Management Review, 51(2), 101-125. https://doi.org/10.2307/41166482

de Neufville, R. (1994). The baggage system at Denver: Prospects and lessons. Journal of Air Transport Management, 1(4), 229-236. https://doi.org/10.1016/0969-6997(94)90014-0

English Standard Version Bible. (2016). Crossway. (Original work published 2001)

Government Accountability Office. (2014). Healthcare.gov: Ineffective planning and oversight practices underscore the need for improved contract management (GAO-14-694). https://www.gao.gov/products/gao-14-694

House of Commons Transport Committee. (2008). The opening of Heathrow Terminal 5 (Twelfth Report of Session 2007-08, HC 543). The Stationery Office.

Ika, L. A. (2009). Project success as a topic in project management journals. Project Management Journal, 40(4), 6-19. https://doi.org/10.1002/pmj.20137

Montealegre, R., & Keil, M. (2000). De-escalating information technology projects: Lessons from the Denver International Airport. MIS Quarterly, 24(3), 417-447. https://doi.org/10.2307/3250968

Serra, C. E. M., & Kunc, M. (2015). Benefits realisation management and its influence on project success and on the execution of business strategies. International Journal of Project Management, 33(1), 53-66. https://doi.org/10.1016/j.ijproman.2014.03.011

Shenhar, A. J., Dvir, D., Levy, O., & Maltz, A. C. (2001). Project success: A multidimensional strategic concept. Long Range Planning, 34(6), 699-725. https://doi.org/10.1016/S0024-6301(01)00097-8

Thomas, W. H. (2015). The basics of project evaluation and lessons learned (2nd ed.). CRC Press.

Williams, T. (2008). How do organizations learn lessons from projects, and do they? IEEE Transactions on Engineering Management, 55(2), 248-266. https://doi.org/10.1109/TEM.2007.912920

Zwikael, O., & Smyrk, J. (2012). A general framework for gauging the performance of initiatives to enhance organizational value. British Journal of Management, 23(S1), S6-S22. https://doi.org/10.1111/j.1467-8551.2012.00823.x