Research Reports

Digital Transformation in Test and Evaluation for AI/ML, Autonomous, and Continuously Evolving Systems

PUBLIC RELEASE
April 2024

COMPLETED
September 2023

AUTHORS: Dr. Laura Freeman, Mr. Geoff Kerr
VIRGINIA TECH

This report, a companion to the AIRC report on Test and Evaluation Methods for Middle-Tier Acquisition, focuses on Pillar 4 of the Director, Operational Test and Evaluation (DOT&E) Implementation Plan: Pioneer T&E of Weapon Systems Built to Change Over Time. The research objective was to explore current industry practice in applying digital engineering methods to test and evaluation of DoD acquired systems and provide recommendations on how to promote further adoption in the T&E community.

Through literature searches, discussions and networking with practitioners, the research team determined that the design and development part of Model-Based Systems Engineering has advanced digital engineering practice beyond that of the T&E portion. To realize the improvement in development timelines, complex system of system capability verification, and operational suitability that the DoD desires, the T&E practices need to be brought to the same level of maturity and integration in the development lifecycle as the other domains of digital engineering. This maturation includes improvements and standardization in ontology, integration of tooling, updates to modeling standards in SysML and other modeling language, improvements in data architecture, and connectivity with existing Modeling and Simulation and Test tracking tooling.

The digital engineering research performed by the AIRC team over the last year has yielded several recommendations for continued focus by DOT&E and AIRC, including pursuing a case study to validate and mature methods and tooling for T&E planning integration with mission engineering/system design; continued work on maturing an ontology focused specifically on digital engineering practices in T&E; a dedicated focus and workshop with DOT&E stakeholders to further explore the use of digital twins for T&E; and further collaboration between DOT&E and AIRC to bring digital engineering short courses to the government T&E offices, as AIRC universities have already developed supporting material on digital engineering that could be leveraged in support of Pillar 4 objectives.