Publications

The literature review, methodology, results, and all relevant publications for the IP2M METRR project are provided on this page for the public’s use.

Report 1 – Forward-looking State-of-the-Art Review on Earned Value Management Systems (EVMS): The Disconnect between Academia and Industry

Abstract: Earned value management (EVM) is a project management approach that can enhance the probability of project success. It is applied widely across different industry sectors (e.g., energy, aerospace, construction, defense, and manufacturing), generally through the use of an earned value management system (EVMS). A holistic and up-to-date literature review on EVM and EVMS does not exist. A literature review can provide a comprehensive perspective on the topic, identifying and summarizing the existing body of knowledge, as a foundation to advance the state of practice of EVM/EVMS. Therefore, the objective of this paper is to investigate the EVM/EVMS state of the art by critically reviewing academic and industry publications, with a specific focus on the maturity of EVMS and the environment surrounding its implementation. By performing a systematic literature review, the authors identified 600 publications since the inception of the EVM concept in 1962, and then narrowed down this list to 160 relevant publications from the last decade for closer review. The findings include the discovery of eight emergent themes. Of these themes, “forecasting/prediction” constitutes the largest portion of the recent literature, followed by “application of EVMS.” One interesting finding is that EVMS maturity, although being a critical topic, is only discussed in one publication. Publications focused on EVM/EVMS have increased in the last decade and significant differences were found between academia and industry literature in terms of the limitations and extensions of EVM/EVMS, EVMS environment, and compliance. A key finding is that designing a reliable EVMS should combine both technical and social aspects of implementation. This forward-looking paper provides a state-of-the-art review while highlighting gaps in the existing EVM/EVMS body of knowledge and introducing new perspectives to support EVMS research and application.

Report 1 Annex A – Full Literature Review Reference List

Executive summary: This document consolidates a list of 433 literature sources dating from 1962 to 2022 and collected as part of the process for developing the Integrated Project/Program Management (IP2M) Maturity and Environment Total Risk Rating (METRR). The consolidated references here may not be an exhaustive list of literature related to Earned Value Management Systems (EVMS) but are all relevant to IP2M METRR.
Earlier findings of the literature review were published by Cho et al. (2020) as part of the American Society of Civil Engineers (ASCE) Construction Research Congress 2020 proceedings. By following a systematic literature review methodology, 160 publications relevant to EVMS covering a diverse set of backgrounds and perspectives from the past decade were identified for closer review. These publications are bolded in this document. The findings of the analysis of the 160 publications were published in the Journal of Management in Engineering by the ASCE in 2022 by Aramali et al. (2022).
The literature list is provided in alphabetical order. This document is part of the deliverables for the research project sponsored by the DOE and has been approved by the research steering committee and Arizona State University (ASU) joint team.

Report 2 – Earned Value Management System State of Practice: Identifying Critical Subprocesses, Challenges, and Environment Factors of a High-Performing EVMS

Abstract: For more than five decades, Earned Value Management Systems (EVMS) have been applied by organizations to manage their projects in different industries. However, many organizations still struggle to apply Earned Value Management (EVM) techniques to assess and reliably control their project outcomes. This work investigated the state of practice of EVMS. This paper identifies challenges facing practitioners, critical EVMS subprocesses, and key EVMS environment factors, based on a large survey of 294 expert respondents. Sample respondents had project management experience of greater than 20 years on average, and represented a diverse set of projects and industries (e.g., capital projects, aerospace, defense, energy, and others). The responses from this survey helped craft an agreed-upon set of definitions for EVM, EVMS, EVMS maturity, and EVMS environment that are all provided in this paper. Moreover, out of 10 total EVMS subprocesses, planning and scheduling was ranked as the subprocess with the greatest impact on EVMS, by far. When ranking the top factors that impact the EVMS environment, the following factors rose to the top: organizational culture; efficient EVMS development process; leadership’s past EVMS experience; effective and accountable leadership; and quality and level of data available. Moreover, leadership attitude toward EVMS was found to be the most critical EVMS challenge, by a wide margin. Comparing different perspectives, the data showed that project/program owners consider EVM implementation costs to be a major challenge, while contractors and consultants consider adequate calendar time needed for preparing EVMS to be a critical EVMS environment factor. Overall, this paper contributes to the engineering management body of knowledge by identifying the most important subprocesses and factors of a high-performing EVMS applied to a diverse array of complex projects and programs (e.g., aerospace, defense, construction, software, etc.) and uncovering corresponding key challenges.

Report 2 Annex A – Survey Analysis Results

Executive summary: To investigate the state of practice of Earned Value Management System (EVMS), a large survey was distributed through external channels targeting participants experienced in EVMS and project controls. The survey consisted of 23 questions, was open for a three-month period (August to October 2019) and received 294 responses from both the public and private sectors. Two hundred and one respondents identified the name of their affiliated employer, indicating that at a minimum, 68 unique organizations participated in the survey as listed in Appendix A. The survey questions included mainly evaluating the definitions for Earned Value Management (EVM) and EVMS, as well as receiving feedback on the impact of the different environment factors and maturity attributes on EVMS. This report lays out the raw results from this survey. Note that in some cases the comments and findings were modified slightly to protect the identity of the respondents. The survey findings formed a strong foundation for the development of the Integrated Project/Program Management (IP2M) Maturity and Environment Total Risk Rating (METRR). This document presents the survey questions and their results. These raw results were also the basis for developing a peer-reviewed journal article, which was published in the Journal of Management in Engineering by the American Society of Civil Engineers (ASCE) in 2021 (Aramali et al. 2021).

Report 3 – An Effective Earned Value Management System (EVMS) is a Team Sport

Abstract: We present a novel framework to assess 27 earned value management system (EVMS) environment factors and investigate their impact on project performance. A study of the EVMS literature, a survey of practitioners, and focus group meetings with 36 EVMS industry and government experts, supported the development of the framework. Focus groups involving 80 practitioners refined and tested it. We analyzed performance data from 35 completed projects/programs representing over US$21.8 billion in
total cost; a project demonstrating a positive environment could save up to 25% in cost versus baseline. Practitioners can develop an effective EVMS environment by following the provided guidance.

Report 3 Annex A – EVMS Environment Assessment Tool Development Process Research Report

Executive summary: This research report summarizes the efforts of the research team to develop the Earned Value Management System (EVMS) environment assessment tool, one of the two components of the Integrated Project/Program Management (IP2M) Maturity and Environment Total Risk Rating (METRR). The authors in conjunction with the research team and an extensive literature review, developed a set of 33 factors to assess the EVMS environment grouped in four categories (Culture, People, Practices, Resources). The authors hosted a series of four industry workshops where 47 industry professionals, representing 24 unique organizations as listed in Appendix A, evaluated the environment factor names and descriptions and helped narrow the list to 27 most important factors that frame the environment within which an EVMS operates. The workshop participants provided comments, and weighted (ranked) the environment factors and categories as elaborated in this report. The collected data was statistically analyzed and used to develop weighted score sheets as a mechanism for environment assessment.

Report 4 – Novel Earned Value Management System Maturity Framework and Its Relation to Project Performance

Abstract: Earned value management systems (EVMS) are used by industry practitioners to successfully manage projects and programs; however, there has been no past study to characterize and evaluate the impact of EVMS maturity on project performance. This paper introduces a novel framework, developed to measure the EVMS maturity, composed of 56 attributes arrayed across ten subprocesses, which together make up an integrated project/program management system. This paper describes how the framework was conceived and developed, based on literature review and data collected from industry workshops. Then, it examines the relationship between EVMS maturity and ten project performance metrics through statistical analyses. The study analyzed project data from 35 projects and programs worth over $21.8 billion in total cost. For this sample, results showed statistically significant differences between projects with high and low EVMS maturity; on average, projects exhibiting high EVMS maturity have 54% less cost overruns and 19% less change orders versus their performance measurement baseline. Higher maturity projects and programs also achieve better customer satisfaction, meet business objectives more often, and are more often in compliance with EVMS standards and guidelines. This paper contributes to the body of knowledge by providing a practical and novel framework for assessing the maturity of earned value management systems. Practitioners can use the framework to improve their integrated project and program management.

Report 4 Annex A – EVMS Maturity Assessment Tool Development Process Research Report

Executive summary: This research report summarizes the efforts of the research team to develop the Earned Value Management System (EVMS) maturity assessment tool, one of the two components of the Integrated Project/Program Management (IP2M) Maturity and Environment Total Risk Rating (METRR). The authors in conjunction with the research team and using an extensive literature review, developed a set of 56 maturity attributes to assess the EVMS maturity. The authors hosted a series of four industry workshops where 56 industry professionals, representing 32 unique organizations as listed in Appendix A, evaluated the maturity attribute names, descriptions, and the narratives of the different maturity levels. The workshop participants provided comments, and weighted (prioritized) the maturity attributes and sub-processes as elaborated in this report. The authors and the research team addressed the comments, and the collected data was statistically analyzed and used to develop weighted score sheets as a mechanism for maturity assessment.

Report 5 – Enhancing Project Success: The Impact of Sociotechnical Integration on Project and Program Management Using Earned Value Management Systems

Abstract: Purpose – Earned value management systems (EVMS), also called integrated project and program management systems, have been greatly examined in the literature, which has typically focused on their technical aspects rather than social. This study aims to hypothesize that improving both the technical maturity of EVMS and the social environment elements of EVMS applications together will significantly impact project performance outcomes. For the first time, empirical evidence supports a strong relationship between EVMS maturity and environment.
Design/methodology/approach –Data was collected from 35 projects through four workshops, attended by
31 industry practitioners with an average of 19 years of EVMS experience. These experts, representing 23
organizations, provided over 2,800 data points on sociotechnical integration and performance outcomes,
covering projects totaling $21.8 billion. Statistical analyses were performed to derive findings on the impact of
technical maturity and social environment on project success.
Findings – The results show statistically significant differences in cost growth, compliance, meeting project
objectives and business drivers and customer satisfaction, between projects with high EVMS maturity and
environment and projects with poor EVMS maturity and environment. Moreover, the technical and social
dimensions were found to be significantly correlated.
Originality/value – Key contributions include a novel and tested performance-driven framework to support
integrated project management using EVMS. The adoption of this detailed assessment framework by
government and industry is driving a paradigm shift in project management of some of the largest and most complex projects in the U.S.; specifically transitioning from a project assessment based upon a binary approach for EVMS technical maturity (i.e. compliant/noncompliant to standards) to a wide-ranging scale (i.e. 0–1,000) across two dimensions.

Report 5 Annex A – Performance Workshops and Data Analysis

Executive summary: This research report summarizes the efforts of the research team to test the IP2M METRR on completed projects and programs. It builds on prior research findings that showed EVMS maturity attributes and environment factors can be defined and measured quantitatively with consistent and weighted score sheets. The main hypothesis in this report focuses on the impact of EVMS maturity and environment on project performance. The authors hosted a series of four workshops where 31 industry professionals representing 23 unique organizations (listed in Appendix A), tested the IP2M METRR tool on 35 projects and programs worth $21.8 Billion USD in installed cost. The participants evaluated 56 maturity attributes and 27 environment factors as applied to each of their reference projects or programs. The collected data was statistically analyzed and used to evaluate the efficacy of the IP2M METRR, and measure the impact of EVMS maturity and environment on project performance. The IP2M METRR was also tested by the authors and research team members on eight additional in-progress projects and programs.

Report 6 – Implementation Guidebook for Using IP2M METRR

Executive summary: This implementation guidebook is part of a series of deliverables for a national research project sponsored by the DOE and has been approved by the research steering committee and Arizona State University (ASU) joint team. It was developed as part of the process for developing the Integrated Project/Program Management (IP2M) Maturity and Environment Total Risk Rating (METRR), pronounced “IP2M meter”. This document introduces the EVMS (or related project management system) attributes and factors, the purpose of the IP2M METRR, and its benefits. The contents of this document can be applied for those situations where the implementation of an EVMS is not required but the use of an integrated project management tool is warranted. This guidebook walks the user through instructions on how to successfully implement the IP2M METRR to assess the effectiveness of a project’s/program’s implementation of the EVMS, with an emphasis of “showing versus telling”. It provides a detailed assessment example and explains the meaning of maturity and environment scores. It also provides strategies on how to implement the IP2M METRR in the user’s organization. In addition, it also includes instructions on how to use the software version of the tool as well as facilitation recommendations. The IP2M METRR tool is provided in the appendices of this guidebook.

Report 6 Annex A – Glossary

Executive summary: This document provides a glossary of terms that was developed as part of the process for developing the Integrated Project/Program Management (IP2M) Maturity and Environment Total Risk Rating (METRR). The terms come from various sources as indicated next to each and as cited in the reference list at the end of this document. The consolidated terms here are not an exhaustive list of terms related to EVMS but are all relevant to IP2M METRR. The glossary list is provided in alphabetical order and related terms are in some cases identified in the write-up of each glossary item.

Report 7 – Employing Technology to Enable Remote Research Charrettes as a Method for Engaging Industry and Uncovering Best Practices: A Novel Approach for a Post-COVID-19 World

Abstract: Methods to collect data in construction engineering and management (CEM) research are evolving, informed by recent technological advancements. One such method is research charrettes that allow effective interactions and knowledge sharing between expert industry practitioners and academic researchers, all colocated in a single venue, enabling rich data collection and live communication. A pivot point in technological evolution occurred with the COVID-19 pandemic, forcing a global shift to remote work. Hence, planned in-person research charrettes had to shift to remote sessions, relying on virtual conferencing platforms and online data collection mechanisms. Technology-enabled charrettes have allowed the authors to collect significantly richer data sets and ensure a more diverse representation of participants, while saving tremendous amounts of time. With the continuing emergence of technological applications, the world might not go back to functioning fully in person. The authors believe remote research charrettes (RRCs) will still be used in a post-COVID-19 world because of their superior performance. This paper builds on a previous publication that described traditional research charrettes as a method to enhance CEM research a decade ago; it offers a significantly updated and improved RRC method based on the knowledge gained from transitioning a dozen in-person charrettes into RRCs. It also presents performance comparisons between RRCs and traditional charrettes by quantifying metrics indicating how RRCs are more time-efficient and cost-saving, harness more participants from more diverse locations, and enable the collection of richer data sets and four times more industry comments and expert feedback. This paper also provides guidance on the integration of technology with traditional research charrettes, hence contributing to the CEM body of knowledge.