Publications

AIRC Perspectives

Best Practices for Addressing New Challenges in Testing and Evaluating Artificial Intelligence Enabled Systems

PUBLISHED
September 2022

AUTHORS: Laura Freeman, Justin Kauffman, Daniel Sobien, Tyler Cody, and Erin Lanus
VIRGINIA TECH 

The integration of artificial intelligence (AI) and statistical machine learning (ML) into complex systems exposes a variety of challenges in traditional test and evaluation (T&E) practices. As more decisions at varying levels are handled by AI-enabled systems (AIES), we need T&E processes that provide a basis for ensuring system effectiveness, suitability, and survivability. 

Measuring the RoI of Digital Engineering: It’s a Journey, Not a Number

PUBLISHED
July 2022

AUTHORS: Tom McDermott1, Kaitlin Henderson2, Eileen Van Aken2, Alejandro Salado3, and Joseph Bradley4
STEVENS INSTITUTE OF TECHNOLOGY 1, VIRGINIA TECH 2, UNIVERSITY OF ARIZONA 3, OLD DOMINION UNIVERSITY 4

Systems engineering as a discipline has long had difficulty providing quantifiable evidence of its value (Honour 2004); DE transformation provides an opportunity to better measure its value. Transitioning from a document-based to a model-based approach is expensive, and organizations want to know if the effort and cost to adopt MBSE is worth it.

Defense Data Grand Prix: Solving Grand Problems to Meet a Grand Vision

PUBLISHED
June 2022

AUTHORS: Stoney Trent1, Hoong Yan See Tao2, Ying Wang2, Paul Grogan2and Christian Lucero1
VIRGINIA TECH1, STEVENS INSTITUTE OF TECHNOLOGY 2

The DoD Data Strategy supports digital modernization and use of data at the speed and scale needed for efficiency and operational advantage. As the future workforce for the DoD, university students studying data science benefit from practical research challenges as part of their education. The Defense Data Grand Prix provides research opportunities that expose students to Defense Acquisition System challenges, decisions, and processes.

Critical Competencies to Support Digital Transformation in Acquisition

PUBLISHED
May 2022

AUTHOR: Nicole Hutchison
STEVENS INSTITUTE OF TECHNOLOGY

As the DoD transitions to digital engineering, there is a need to develop and maintain an acquisition workforce that is literate in model-based approaches, competent in digital models, methods, and tools, and understands digital artifacts across the acquisition lifecycle. This perspective paper explores those competencies and levels of proficiency needed to support digital transformation.

Managing Portfolios in an Environment of Dynamic Requirements and Uncertainty

PUBLISHED
April 2022

AUTHORS: William B. Rouse1, Dinesh Verma2, Thomas A. McDermott, Jr.2, and Philip S. Antón2
GEORGETOWN UNIVERSITY 1, STEVENS INSTITUTE OF TECHNOLOGY 2

Organizations would like to have the flexibility and agility to address the uncertainty of both needs and technologies that results from challenges related to performance, organizational experience, supply chains, etc. This perspective paper outlines and illustrates new ways of thinking and allocating resources that organizations can adopt to address these challenges.

CHALLENGES TO INNOVATION TRANSITION: The Valley of Death Results from More Than a Lack of Flexible Funding

PUBLISHED
March 2022

AUTHOR: Philip S. Antón
STEVENS INSTITUTE OF TECHNOLOGY

Innovation is appealing and attractive, but it is a difficult and risky business. The Department of Defense (DoD) and the commercial sector have long struggled with transitioning and fielding innovative concepts and systems to operational use. Data show that very few innovations are successfully fielded. To tackle innovation, the DoD and its partners have created a range of organizations to pursue and apply innovative concepts.

Research Reports

Thoughts on Agility Through Appropriations

PUBLISHED
October 2022

AUTHOR: Philip S. Antón
STEVENS INSTITUTE OF TECHNOLOGY

The U.S. Department of Defense (DoD) seeks sufficient funds from the U.S. Congress to meet critical national security missions. However, predicting, budgeting, authorizing, and appropriating the type and amount of needed funding can be challenging, especially when threats and technologies are changing faster than Planning, Programming, Budgeting, and Execution (PPBE) process timelines, which run over two years given that the DoD starts planning 19 months before the programming and budgeting stage begins. Threats and needs can even change after budgets are authorized and appropriated by Congress and reprogramming of those funds are limited. This paper hypothesizes different approaches related to budgeting and appropriating to improve defense acquisition given inherent DoD and Congressional equities.

Additive Manufacturing and Digital Engineering Strategy Development

PUBLISHED
October 2022

AUTHORS: Dr. Daniel A. DeLaurentis, Dr. Jitesh H. Panchal
PURDUE UNIVERSITY

The overall objective of this research was to develop tools and knowledge to support the development of an additive manufacturing supply chain sustainment strategy and a digital engineering strategy. As we initiated the research effort at the beginning, we asked ourselves: Can the whole supply chain and sustainment strategy change given the additive manufacturing opportunities instead of traditional manufacturing? In what timeframe? What are the limits (e.g., materials science; systems engineering; cost; reliability; infrastructure)? To effectively perform mission engineering and ensure readiness for additive manufacturing, we set up a digital environment ready for sustainment and have the capabilities we need when we need them.

Data-Driven Capability Portfolio Management Pilot

PUBLISHED
October 2022

AUTHORS: Dr. Daniel A. DeLaurentis, Dr. Jitesh H. Panchal
PURDUE UNIVERSITY

AIRC research adapted a previously developed system-of-systems analytic workbench (SoS-AWB) of analytic tools to create a decision-support prototype effective for informing decisions in Integrated Acquisition Portfolio Reviews (IAPRs). Mission engineering analysis and portfolio optimization of an anti-surface warfare (ASuW) mission thread using personnel and munitions in the surface, aviation, and space domains was demonstrated via the prototype software. These advanced prototypes provide a broader range of insights (e.g., resource tradeoffs, cost-sensitivity analysis, and the most robust ASuW systems to be acquired in specific portfolios) for stakeholder decision making.

Joint Capabilities Integration and Development System (JCIDS)

PUBLISHED
September 2022

AUTHORS: Mo Mansouri, Michael McGrath, Donald Schlomer, Dinesh Verma, and Philip S. Antón
STEVENS INSTITUTE OF TECHNOLOGY

The Joint Capabilities Integration and Development System (JCIDS) in the U.S. Department of Defense (DoD) has come under increasing scrutiny over the past decade, with a particular focus on its latencies. Since a validated requirement is needed before an acquisition program can begin, delays in the JCIDS process were perceived by Congress as adding months or even years to the acquisition cycle. In support of the DoD’s response to Congressional concerns, the Acquisition Innovation Research Center (AIRC) was asked to assess how it can improve the efficiency of developing and approving capability requirements and to develop a model to show the effects of proposed alternatives.

The Nature of the Defense Innovation Problem

PUBLISHED
July 2022

AUTHORS: Melissa Flagg, Molly Nadolski, Jose Sanchez, Tom McDermott, and Philip S. Antón
STEVENS INSTITUTE OF TECHNOLOGY

We identified the following recommendations and next steps that could begin moving the DoD more towards empowering, creating, identifying, assessing, and adopting operationally significant innovation (rather than attractive innovation that cannot be justified and pursued given limited resources).

IP Evaluation and Valuation Study

PUBLISHED
October 2021

AUTHORS: Claudia Stewart, Karrie Wessing, Michael Ryan, Richard Beutel, and Sonia Kaestner
GEORGETOWN UNIVERSITY

This study develops lean and appropriately customizable Intellectual Property (IP) management modalities that can be fully integrated into acquisition and product support strategies; acquire only the necessary deliverables and license rights at the appropriate times to secure DoD’s interests throughout the product life cycle.

IP Data Repository

PUBLISHED
October 2021

AUTHORS: Laura Freeman and Brian Mayer
VIRGINIA TECH

Tasked to develop a data and electronic information repository for models, samples, templates, and exemplars of Intellectual Property (IP), through the use of the “FAIR” principles (Findability, Accessibility, Interoperability, and Reusability), the team researched government data using a two-tiered approach that explored the requested data fields in the AIRC Performance Work Statement (PWS) and publicly available metadata to create a longitudinal topology of DoD contracting activity.

Incubator Reports

Innovation for Digital Transformation and Policy Analytics

PUBLISHED
November 2022

AUTHORS: Dr. Azad M. Madni, Dr. Dan Erwin
UNIVERSITY OF SOUTHERN CALIFORNIA

In this study, led by Principal Investigator Dr. Azad M. Madni of the University of Southern California, a risk-based validation approach was developed and applied to the acquisition of an Unmanned Aerial Vehicle (UAV) to detect forest fires. The concept of risk was defined within the context of system acquisition and defined an initial validation metamodel. The system comprises a UAV with a GPS and two cameras that can send photos to a processing facility. The validation method was based on the detection of hot spots and smoke. The central idea was to build a tree to help evaluate what-if analyses, look for overlaps and gaps, and help prioritize validation demonstrations.

Cognitive Training Assistant for Cost Estimators

PUBLISHED
November 2022

AUTHORS: Dr. Daniel Selva1, Dr. Theodora Chaspari1, Dr. Alejandro Salado2
TEXAS A&M UNIVERSITY 1, UNIVERSITY OF ARIZONA 2

Cognitive assistants (CAs) enable humans to solve complex tasks and problems more efficiently. The goal of this project, led by Principal Investigator (PI) Dr. Daniel Selva (Texas A&M University), was to demonstrate proof of concept for a CA that would assist in training new cost estimators in the Department of Defense (DoD). This CA was developed to provide trainees with interactive, hands-on opportunities to learn concepts, methods, and best practices related to estimating the lifecycle costs of complex systems.

Calculating Return on Investment in a Department of Defense Context

PUBLISHED
October 2022

AUTHORS: Dr. Eric W. Burger, Dr. Robin Dillon-Merrill, Erika Heeren-Moon, Franco Forti
GEORGETOWN UNIVERSITY

To determine the value of expenditures, businesses use return on investment (ROI) to measure success over time and eliminate the guesswork of future business decisions. The ability to calculate ROI can be beneficial to any business, regardless of its size or industry. Although the Department of Defense (DoD) uses business analytics in its acquisition process, including market research studies and alternative analyses, it does not aim to maximize profits. The DoD instead seeks to increase operational capabilities, make life safer, establish deterrent positions and reduce risk. This research aims to identify opportunities in the DoD acquisition process where ROI calculations can improve the performance of defense acquisitions in cutting-edge technology. The research team, led by Principal Investigator Dr. Eric W. Burger of Georgetown University, believes that the DoD will be able to accelerate contract development, letting and awarding with higher functionality, and lower total cost by adopting best practices of the private sector and non-DoD public sector ROI calculations.  

Management and Business Knowledge Representation for Decision Making

PUBLISHED
October 2022

AUTHOR: Dr. Johnathan Mun
NAVAL POSTGRADUATE SCHOOL

This current first-phase research looks at how the Department of Defense (DoD) can inculcate institutional corporate memory. Specifically, the research tests and develops recommendations about how a transparent Decisions Options Register (DOR) integrated intelligent database system can be developed, where the DOR helps capture all historical decisions (assumptions, data inputs, constraints, limitations, competing objectives, and decision rules) for programs within the Department of Defense (DoD).

Incentives for Motivating Workforce Agility and Innovation

PUBLISHED
September 2022

AUTHORS: Dr. Amanda Girth, Dr. Mike Rayo, Dr. Elizabeth Newton, Mr. Asher Balkin, Ms. Christine Jefferies, Ms. Priyanka Tewani
THE OHIO STATE UNIVERSITY

This pilot study, conducted in partnership with the Air Force Installation Contracting Center (AFICC), developed approaches and initial set of systemic pressures that hinder innovative behavior in the acquisition workforce of the Department of Defense (DoD). An approach was developed to create interventions that mitigate systemic pressures to incentivize lasting behavioral and cultural changes to improve acquisition outcomes in support of the National Defense Strategy.

Annual Report

Annual Report for FY 2021 Extramural Acquisition Innovation and Research Activities

PUBLISHED
September 2021

The Secretary of Defense created the Acquisition Innovation Research Center (AIRC) on September 30, 2020, in response to section 2361(a) of title 10 U.S. Code. AIRC has achieved its goals for its inaugural year, and a breadth of activities have been funded to engage AIRC on key challenges facing the Department of Defense (DoD).